Communication system, communication device, seating-order determination device, communication method, recording medium, group-determination-table generating method, and group-determination-table generating device

ABSTRACT

In a communication system having at least three communication devices communicating with each other, a seating-order determination device for generating seating-order information for information sent from each communication device and for transmitting the seating-order information to each communication device is provided. Each communication device controls the output position of each information according to the seating-order information to output the information sent from the other communication devices in a seating order corresponding to the seating-order information. The seating order is always automatically changed to the most appropriate condition according to the progress of a conference and the state of conversations to provide the user with a comfortable conference environment and a comfortable communication environment.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to communication systems in whichcommunications are performed among three or more communication devices,the communication devices which constitute the communication systems,and seating-order determination devices. The present invention furtherrelates to group-determination-table generation devices for generating agroup determination table used in the seating-order determinationdevices and the communication methods.

2. Description of the Related Art

In conventional teleconference systems, images and sound generated at aplurality of conference rooms distant from each other are transferredamong the conference rooms through a network, and the images and soundsent from the other conference rooms are reproduced at each conferenceroom. This allows a conference to be held as if all participants sataround one table.

In these conventional teleconference systems, conference participantsare allowed to talk at the same time in each conference room.

In an actual conference, in some cases, some conversation groups aremade among participants according to the situation of a progress in theconference, and topics are different in the groups. Each group is madetentatively by a part of conference participants for a certain topic,and the group is flexible. More specifically, a conversation group maybe made according to a progress in the conference; persons constitutinga group may be changed; and one group may be further divided into aplurality of groups. The structures of groups are always being changed.

In the conventional teleconference systems, each participant sees theother participants on monitor devices. Each monitor device correspondsto one participant, and their relationship is fixed. In a conferencehaving six participants, for example, each participant sees the otherfive participants on five monitor devices. Each monitor device displaysan assigned participant. In other words, in a teleconference terminalused by one participant, five monitor devices MDa, MDb, MDc, MDd, andMDe display the other participants HMa, HMb, HMc, HMd, and HMe in analways fixed manner.

Assuming that each monitor device is handled as the seat of theparticipant corresponding to the monitor device, it can be consideredthat the order of seats (seating order) can be made changeable to someextent.

In other words, the five monitor devices MDa, MDb, MDc, MDd, and MDearranged in this order do not display the participants HMa, HMb, HMc,HMd, and HMe in a fixed manner, and the relationship between the monitordevices and the participants is made changeable.

With this feature, when a group is formed among participants, theseating order can be changed according to the formed group. When theuser of the terminal and the participants HMb and HMd form a group, forexample, if the seating order is changed such that the monitor deviceMDa displays the participants HMb and the monitor device MDb displaysthe participants HMd, the group is made to have a convenient conditionfor their conversation.

The seating order is changed, for example, by the seating-order-changingoperations of the user. It is not realistic that the user performs aseating-order changing operation according to a group formed or releasedduring a conference. This means that the user needs to perform a verytroublesome operation. In addition, especially during a conference, theuser wants to concentrate on the conference without performing anyoperations. Furthermore, a terminal user is not necessarily familiarwith operations.

It can be considered that a system operator is assigned toseating-order-changing operations. It is also unrealistic, however,because extra man-power is required, and an operator usually cannotcorrectly understand conversation groups which always change theirparticipants.

With the above-described reasons, even in a system which allows aseating order to be changed, the feature is usually not utilized.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the foregoingpoints. It is an object of the present invention to provide acommunication system, such as a teleconference system, a communicationdevice, a seating-order determination device, a communication method, arecording medium, a group-determination-table generating method, and agroup-determination-table generating device, in which a seating ordercan be appropriately changed according to conversations which flexiblyprogress among participants to provide a more suitablecommunication-conversation environment.

The foregoing object is achieved in one aspect of the present inventionthrough the provision of a communication system including at least threecommunication devices, and a seating-order determination device forgenerating seating-order information at each point of time forinformation sent from each communication device and for transmitting theseating-order information to each communication device.

Each communication device may control the output position of theinformation sent from other communication devices, according to theseating-order information to output the information sent from the othercommunication devices in a seating order corresponding to theseating-order information.

In this case, the seating order is always automatically changed to themost appropriate condition according to the progress of a conference andthe state of conversations to provide the user with a suitableconference environment and a suitable communication environment.

The seating-order determination device may generate the seating-orderinformation for the information sent from each communication deviceaccording to the degree of attention which the user of eachcommunication device pays to the information sent from eachcommunication device. In this case, the most appropriate seating orderis implemented.

The seating-order determination device may group the information sentfrom each communication device according to the degree of attentionwhich each user pays to the information sent from each communicationdevice, and generate the seating-order information according to theresult of grouping. In this case, the seating order is automaticallychanged with conversation groups being taken into account and a suitableteleconference system is implemented.

The seating-order information may be generated such that informationbelonging to the same group is arranged. In this case, the seating orderis changed so as to collect the members of groups.

The seating-order information may also be generated such thatinformation belonging to the same group is dispersed almost uniformly.In this case, the user is provided with an easy-to-converse environment.

When the seating order is changed according to the seating-orderinformation, each communication device may output indication informationindicating a change in the seating order to the user, for example, byimage information or audio information. In this case, the user canunderstand that the seating order is to be changed, and the user isprevented from confusing with a change in the seating order.

When the seating order is changed according to grouping, eachcommunication device may output indication information indicating thestate of grouping to the user. In this case, the user successfullyunderstands the states of conversation groups. This indicationinformation also helps the user understand the current condition. As theindication information, a background image color may be used, which isone of the most easy-to-understand indications.

The degree of attention may be determined according to user-behaviordetection information or information specified by the user. In thiscase, the degree of attention is suited as a reference for a change inthe seating order. More specifically, when the user-behavior detectioninformation includes the lines of sight or a face direction of the user,most suitable control is implemented by the use of a natural operationof the user.

The grouping may be performed according to the statistical relationshipbetween a group structure and the degree of attention which the user ofeach communication device pays to the information sent from the othercommunication devices. In this case, the grouping is suited to generatethe seating-order information.

Therefore, when a group-determination-table generating method and agroup-determination-table generating device according to the presentinvention hold the statistical relationship, a suitable operation isimplemented for changing the seating order.

The foregoing object is achieved in another aspect of the presentinvention through the provision of a seating-order determination deviceprovided for a communication system having at least three communicationdevices, including seating-order-information generating means forgenerating seating-order information at each point of time forinformation sent from each communication device; and transmitting meansfor sequentially transmitting the seating-order information generated bythe seating-order-information generating means to each communicationdevice.

The foregoing object is achieved in still another aspect of the presentinvention through the provision of a communication device in acommunication system including at least three communication devicescommunicating with each other, including receiving means for receivinginformation and seating-order information sent from other communicationdevices; attention-degree-information generating means for detecting thedegree of attention which the user pays to the information sent from theother communication devices to generate attention-degree information;transmitting means for transmitting the attention-degree informationgenerated by the attention-degree-information generating means;presenting means for presenting the information sent from the othercommunication devices; and information manipulation and distributionmeans for controlling the output positions of the information sent fromthe other communication devices according to the seating-orderinformation received by the receiving means to output the informationsent from the other communication devices in a seating ordercorresponding to the seating-order information.

The foregoing object is achieved in yet another aspect of the presentinvention through the provision of a group-determination-tablegenerating device for generating a group determination table used forgrouping information sent from each communication device according tothe degree of attention of the user of each communication device in acommunication system having at least three communication devices,including statistics means for reading attention-degree patternsindicating the degree of attention which the user of each communicationdevice pays to information sent from other communication devices andgroup structure patterns indicating the group state of each user, andfor collecting statistics on the attention-degree patterns and the groupstructure patterns; determination means for determining thecorrespondence between each attention-degree pattern and one of thegroup structure patterns from the statistics obtained by the statisticsmeans; and determination-table generating means for generating a groupdetermination table indicating attention-degree patterns and groupstructure patterns for which the correspondence is determined by thedetermination means.

The foregoing object is achieved in yet still another aspect of thepresent invention through the provision of a communication method for acommunication system having at least three communication devices,including a seating-order generating step of generating seating-orderinformation at each point of time for information sent from eachcommunication device; and a transmitting step of sequentiallytransmitting the seating-order information generated in theseating-order generating step to each communication device.

The foregoing object is achieved in a further aspect of the presentinvention through the provision of a seating-order determination methodfor a seating-order determination device provided for a communicationsystem having at least three communication devices, including aseating-order-information generating step of generating seating-orderinformation at each point of time for information sent from eachcommunication device; and a transmitting step of sequentiallytransmitting the seating-order information generated in theseating-order-information generating step to each communication device.

The foregoing object is achieved in a still further aspect of thepresent invention through the provision of a communication method for acommunication device in a communication system including at least threecommunication devices communicating with each other, including areceiving step of receiving information and seating-order informationsent from other communication devices; an attention-degree-informationgenerating step of detecting the degree of attention which the user paysto the information sent from the other communication devices to generateattention-degree information; a transmitting step of transmitting theattention-degree information generated in theattention-degree-information generating step; a presenting step ofpresenting the information sent from the other communication devices;and an information manipulation and distribution step of controlling theoutput positions of the information sent from the other communicationdevices according to the seating-order information received in thereceiving step to output the information sent from the othercommunication devices in a seating order corresponding to theseating-order information.

The foregoing object is achieved in a yet further aspect of the presentinvention through the provision of a group-determination-tablegenerating method for a group-determination-table generating device forgenerating a group determination table used for grouping informationsent from each communication device according to the degree of attentionof the user of each communication device in a communication systemhaving at least three communication devices, including a statistics stepof reading attention-degree patterns indicating the degree of attentionwhich the user of each communication device pays to information sentfrom other communication devices and group structure patterns indicatingthe group state of each user, and of collecting statistics on theattention-degree patterns and the group structure patterns; adetermination step of determining the correspondence between eachattention-degree pattern and one of the group structure patterns fromthe statistics obtained in the statistics step; and adetermination-table generating step of generating a group determinationtable indicating attention-degree patterns and group structure patternsfor which the correspondence is determined in the determination step.

The foregoing object is achieved in a yet still further aspect of thepresent invention through the provision of a recording medium forstoring a processing program related to seating information forinformation sent from each communication device in a communicationsystem having at least three communication devices, the processingprogram including a seating-order generating step of generatingseating-order information at each point of time for information sentfrom each communication device; and a transmitting step of sequentiallytransmitting the seating-order information generated in theseating-order generating step to each communication device.

The foregoing object is achieved in an additional aspect of the presentinvention through the provision of a recording medium for storing aprocessing program related to seating-order determination in aseating-order determination device provided for a communication systemhaving at least three communication devices, the processing programincluding a seating-order-information generating step of generatingseating-order information at each point of time for information sentfrom each communication device; and a transmitting step of sequentiallytransmitting the seating-order information generated in theseating-order-information generating step to each communication device.

The foregoing object is achieved in a still additional aspect of thepresent invention through the provision of a recording medium forstoring a processing program related to communication in a communicationdevice of a communication system including at least three communicationdevices communicating with each other, the processing program includinga receiving step of receiving information and seating-order informationsent from other communication devices; an attention-degree-informationgenerating step of detecting the degree of attention which the user paysto the information sent from the other communication devices to generateattention-degree information; a transmitting step of transmitting theattention-degree information generated in theattention-degree-information generating step; a presenting step ofpresenting the information sent from the other communication devices;and an information manipulation and distribution step of controlling theoutput positions of the information sent from the other communicationdevices according to the seating-order information received in thereceiving step to output the information sent from the othercommunication devices in a seating order corresponding to theseating-order information.

The foregoing object is achieved in a yet additional aspect of thepresent invention through the provision of a recording medium forstoring a processing program related to group-determination-tablegeneration in a group-determination-table generating device forgenerating a group determination table used for grouping informationsent from each communication device according to the degree of attentionof the user of each communication device in a communication systemhaving at least three communication devices, the processing programincluding a statistics step of reading attention-degree patternsindicating the degree of attention which the user of each communicationdevice pays to information sent from other communication devices andgroup structure patterns indicating the group state of each user, and ofcollecting statistics on the attention-degree patterns and the groupstructure patterns; a determination step of determining thecorrespondence between each attention-degree pattern and one of thegroup structure patterns from the statistics obtained in the statisticsstep; and a determination-table generating step of generating a groupdetermination table indicating attention-degree patterns and groupstructure patterns for which the correspondence is determined in thedetermination step.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a teleconference system according to anembodiment of the present invention.

FIG. 2 is a block diagram of a teleconference device according to theembodiment.

FIG. 3 is a view showing a method for generating attention-degreeinformation according to the embodiment.

FIG. 4 is a view showing another method for generating attention-degreeinformation according to the embodiment.

FIG. 5 is a block diagram of a seating-order determination deviceaccording to the embodiment.

FIG. 6 is a view showing the initial state of an attention-destinationtable according to the embodiment.

FIG. 7 is a view showing the initial state of a group table according tothe embodiment.

FIG. 8 is a flowchart of initialization processing applied to theattention-destination table and the group table according to theembodiment.

FIG. 9 is a flowchart of processing to be performed whenattention-degree information is generated according to the embodiment.

FIG. 10 is a view showing an attention-destination table to whichattention-degree information has been input according to the embodiment.

FIG. 11 is a flowchart of processing for inputting the contents of theattention-destination table to the group table according to theembodiment.

FIG. 12 is a view showing a group table obtained during the processingaccording to the embodiment.

FIG. 13 is a view showing an attention-destination table obtained duringthe processing according to the embodiment.

FIG. 14 is a view showing a group table obtained during the processingaccording to the embodiment.

FIG. 15 is a view showing an attention-destination table obtained duringthe processing according to the embodiment.

FIG. 16 is a view showing a group table obtained during the processingaccording to the embodiment.

FIG. 17 is a view showing an attention-destination table obtained duringthe processing according to the embodiment.

FIG. 18 is a view showing a group table obtained during the processingaccording to the embodiment.

FIG. 19 is a view showing an attention-destination table obtained duringthe processing according to the embodiment.

FIG. 20 is a view showing a group table obtained during the processingaccording to the embodiment.

FIG. 21 is a view showing an attention-destination table obtained duringthe processing according to the embodiment.

FIG. 22 is a view showing a group table obtained during the processingaccording to the embodiment.

FIG. 23 is a view showing an attention-destination table obtained duringthe processing according to the embodiment.

FIG. 24 is a view showing a group table obtained during the processingaccording to the embodiment.

FIG. 25 is a view showing an attention-destination table obtained duringthe processing according to the embodiment.

FIG. 26 is a flowchart of seating-order determination processingaccording to the embodiment.

FIG. 27A and FIG. 27B are views showing example seating-orderdetermination according to the embodiment.

FIG. 28A, FIG. 28B, and FIG. 28C are views showing another exampleseating-order determination according to the embodiment.

FIG. 29 is a view showing example distributed-seating-orderdetermination according to the embodiment.

FIG. 30 is a block diagram of an information manipulation anddistribution section according to the embodiment.

FIG. 31 is a block diagram of an image manipulation section according tothe embodiment.

FIG. 32 is a block diagram of an audio manipulation section according tothe embodiment.

FIG. 33 is a block diagram of an information distribution sectionaccording to the embodiment.

FIG. 34A and FIG. 34B are views showing example image processing appliedwhen the seating order is changed according to the embodiment.

FIG. 35A, FIG. 35B, FIG. 35C, and FIG. 35D are views showing anotherexample image processing applied when the seating order is changedaccording to the embodiment.

FIG. 36A, FIG. 36B, and FIG. 36C are views showing still another exampleimage processing applied when the seating order is changed according tothe embodiment.

FIG. 37A and FIG. 37B are views showing example image processing forindicating groups according to the embodiment.

FIG. 38 is a view showing group patterns according to the embodiment.

FIG. 39 is a view showing the frequency table of group patterns forattention patterns according to the embodiment.

FIG. 40 is a view showing a group determination table according to theembodiment.

FIG. 41 is a view showing an attention-pattern conversion tableaccording to the embodiment.

FIG. 42 is a view showing a group conversion table according to theembodiment.

FIG. 43 is a view showing a representative frequency table according tothe embodiment.

FIG. 44 is a view showing a representative-group determination tableaccording to the embodiment.

FIG. 45 is a view showing a group inverted-conversion table according tothe embodiment.

FIG. 46A and FIG. 46B are views showing group conversion methodsaccording to the embodiment.

FIG. 47 is a functional block diagram of a group-determination-tablegenerating device according to the embodiment.

FIG. 48 is a flowchart of processing for generating the representativefrequency table according to the embodiment.

FIG. 49 is a flowchart of processing for generating therepresentative-group determination table according to the embodiment.

FIG. 50 is a flowchart of processing for generating the groupdetermination table according to the embodiment.

FIG. 51 is a view showing a satisfaction-degree weight table accordingto the embodiment.

FIG. 52 is a view showing example degrees of attentions paid toparticipants according to the embodiment.

FIG. 53 is a view showing the names of seats according to theembodiment.

FIG. 54 is a view showing an example seating order according to theembodiment.

FIG. 55 is a flowchart of sight-line detection processing according tothe embodiment.

FIG. 56 is a view showing detection of both-end positions of an eyeaccording to the embodiment.

FIG. 57 is a view showing a nostril-position detection area according tothe embodiment.

FIG. 58 is a view showing the both-end positions of eyes, nostrilpositions, and eyeball-center positions according to the embodiment.

FIG. 59 is a view showing detection of a sight-line direction accordingto the embodiment.

FIG. 60 is a view showing a method for obtaining a line which makes thesecondary moment of a predetermined set of pixels minimum, according tothe embodiment.

FIG. 61 is a flowchart of processing for detecting a face directionaccording to the embodiment.

FIG. 62A and FIG. 62B are views showing original images used fordetecting a face direction according to the embodiment.

FIG. 63A and FIG. 63B are views showing hair areas and skin areas usedfor detecting a face direction according to the embodiment.

FIG. 64A and FIG. 64B are views showing the centers of gravity of thehair areas and of the skin areas according to the embodiment.

FIG. 65 is a view showing an example relationship between a differenceand an angle during face-direction detection according to theembodiment.

FIG. 66 is an outlined internal view of a specific monitor device,viewed from a side thereof, according to the embodiment.

FIG. 67 is an outlined elevation of the specific monitor deviceaccording to the embodiment.

FIG. 68 is a block diagram of an actual structure implementing a signalprocessing device and a seating-order determination device in eachteleconference device in the teleconference system according to theembodiment.

FIG. 69 is a view showing an outlined structure of anotherteleconference device which displays conference participants on a screenand disposes sound images by speakers according to the embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENT

A teleconference system according to an embodiment of the presentinvention will be described below in the following order.

1. Structure of communication system

2. Structure of teleconference device

3. Structure of seating-order determination device

4. Grouping processing in the seating-order determination device

5. Seating-order determination operation through grouping in theseating-order determination device

6. Seating-order changing processing performed according toseating-order information in a teleconference device

7. First example of grouping processing which uses a statisticalrelationship in the seating-order determination device

8. Second example of grouping process which uses a statisticalrelationship in the seating-order determination device

9. Seating-order determination operation not through grouping in theseating-order determination device

10. Attention-degree-information generating operation in ateleconference device

11. Structure of monitor device

12. Example structure of each device

1. Structure of Communication System

FIG. 1 shows an outlined structure of a teleconference system accordingto an embodiment of the present invention. In the present specification,a system refers to the whole structure formed of a plurality of devicesand sections.

In the teleconference system shown in FIG. 1, teleconference devicesTCD1 to TCDn are assigned to conference participants HM1 to HMn locatedat a plurality of (one to n) positions. The teleconference devices TCD1to TCDn are connected through a communication network NT formed, forexample, of an integrated services digital network (ISDN).

When the conference participants HM1 to HMn do not need to bedistinguished from each other, they are hereinafter collectively calledconference participants HM. In the same way, when the teleconferencedevices TCD1 to TCDn do not need to be distinguished from each other,they are hereinafter collectively called teleconference devices TCD. InFIG. 1, an IDSN is taken as an example of the communication network NT.Instead of an ISDN, other transfer media, such as cable televisionnetworks, the Internet, and digital satellite communication, can beused.

Each teleconference device TCD captures the image data and audio data ofthe corresponding conference participant HM; performs mutualcommunication with other teleconference devices TCD through thecommunication network NT; and can reproduce the image data and audiodata (can monitor the images and sound) of the other conferenceparticipants HM sent from the other teleconference devices TCD.

2. Structure of Teleconference Device

Each teleconference device TCD constituting the teleconference systemhas a structure shown in FIG. 2.

The teleconference devices TCD1 to TCDn have the same structure. FIG. 2shows a detailed example structure of the teleconference device TCD1 asa representative of the plurality of teleconference devices TCD1 toTCDn.

The teleconference device TCD1 includes at least a signal processingdevice SPD1 connected to the communication network NT, for transmittingand receiving signals to and from the other teleconference devices TCD2to TCDn which constitute the teleconference system and for applyingsignal processing described later to signals transmitted and received;and monitor devices MD2 to MDn in which the image data and audio data ofthe conference participants HM2 to HMn transmitted from the otherteleconference devices TCD2 to TCDn constituting the teleconferencesystem can be monitored correspondingly to the teleconference devicesTCD2 to TCDn.

When the monitor devices MD2 to MDn do not need to be distinguished fromeach other, they are hereinafter collectively called monitor devices MD.

The users of the teleconference devices TCD1 to TCDn are fixed to theconference participants HM1 to HMn. The relationships between themonitor devices MD in the teleconference devices and the information ofconference participants HM displayed thereon are not fixed butdynamically changed according to seating-order information describedlater.

For simplicity, until a change of a seating order is described, adescription will be given under the assumption that the monitor devicesMD1 to MDn correspond to the conference participants HM1 to HMn locatedin the teleconference devices TCD1 to TCDn, respectively.

The signal processing device SPD1 of the teleconference device TCD1includes a network connection terminal TN1 for connecting to thecommunication network NT; an information transmitting and receivingsection TRB1 for transmitting and receiving information to and from thecommunication network NT; an information manipulation and distributionsection PB1 for applying information manipulation and distributionprocessing described later to signals to be sent to monitor device MD2to MDn; an attention-degree-information generating section JB1 forgenerating attention-degree-information used for dynamically changing aseating order during a conference, as described later; output terminalsTO2 to TOn for outputting signals separately to the monitor devices MD2to MDn; input terminals TI2 to TIn for receiving signals separately fromthe monitor devices MD2 to MDn; and an input terminal TS for receiving asignal from a switch SW, which generates switch-pressing informationdescribed later, used for generating the attention-degree information.

A detailed structure of each of the monitor devices MD2 to MDn will bedescribed later. Each monitor device MD includes, as main components, atleast a speaker provided at the front side of the body of the monitordevice, and a display section disposed such that its screen G isdirected in a predetermined direction (such as a direction towards theparticipant HM1).

At least one monitor device MD among the monitor devices MD2 to MDn isprovided with a microphone for capturing sound around the teleconferencedevice TCD1 and the sound of what the conference participant HM1 says,and a camera (such as a video camera) for capturing an image of theconference participant HM1.

It is preferable that a monitor device provided with a microphone and acamera be disposed at the position of a monitor device (monitor deviceMDm in the case shown in FIG. 2) which directly faces the conferenceparticipant HM1. It is also possible that all the monitor devices MD2 toMDn are provided with microphones and cameras.

Image data captured by the camera and audio data captured by themicrophone in the monitor device MD are transmitted to the otherteleconference devices TCD2 to TCDn through the signal processing deviceSPD1 and the communication network NT.

Images based on image data sent from the teleconference devices TCD2 toTCDn are displayed on the display sections of the monitor devices MD2 toMDn, and sound based on audio data sent from the teleconference devicesTCD2 to TCDn is output from the speakers of the monitor devices.

In other words, the monitor devices MD2 to MDn correspond to theteleconference devices TCD2 to TCDn with one-to-one correspondence. Forexample, images based on image data (image data of the conferenceparticipant HM2 and the surrounding thereof) captured by the camera ofthe teleconference device TCD2 and sent through the communicationnetwork NT are displayed on the screen G of the display section of themonitor device MD2, and sound based on audio data (audio data of whatthe conference participant HM2 says) captured by the microphone of theteleconference device TCD2 and sent through the communication network NTis output from the speaker of the monitor device MD2.

In the same way, images based on image data captured by the camera ofthe teleconference device TCD3 and transmitted are displayed on thescreen of the display section of the monitor device MD3, and sound basedon audio data captured by the microphone of the teleconference deviceTCD3 and transmitted is output from the speaker of the monitor device.The other monitor devices MD work in the same way. Images sent fromteleconference devices TCD are displayed and sound is output.

As described above, however, the relationships between the monitordevices MD2 to MDn and the teleconference devices TCD2 to TCDn are notfixed but is dynamically changed as a so-called seating-order change.Therefore, the above-described one-to-one correspondence relationship isa tentative correspondence relationship such as that used in a systeminitial condition.

It can be considered that image data transmitted and received throughthe communication network NT among the teleconference devices TCD1 toTCDn includes still-picture data as well as motion-picture data.

The monitor devices MD2 to MDn are disposed as shown in FIG. 2 as if theconference participant HM1, who is in a conference room having theteleconference device TCD1, and the other conference participants HM2 toHMn (those displayed on the display sections of the monitor devices MD2to MDn) were around one table to have a conference. Assuming that sixteleconference devices TCD are used in the teleconference system andeach teleconference device TCD is provided with five monitor devices,the five monitor devices can be disposed as shown in the figure suchthat a conference participant HM and the five monitor devices MD form,for example, a regular hexagon.

The attention-degree-information generating section JB of the signalprocessing device SPD in each teleconference device TCD generatesattention-degree information used when a seating order is dynamicallychanged during a conference, as described below.

The attention-degree-information generating section JB1 of the signalprocessing device SPD1 in the teleconference device TCD1 is taken as anexample among the attention-degree-information generating sections JB1to JBn corresponding to the teleconference devices TCD1 to TCDn, and itsoperation will be described below.

The attention-degree-information generating section JB1 detects thedegrees of attention which the conference participant HM1 pays to theother conference participants according to image data sent from thecamera, for example, of the monitor device MDm disposed in the front ofthe conference participant HM1, and generates attention-degreeinformation used for dynamically changing the seating order, asdescribed later, according to the result of detection.

The degrees of attention which the conference participant HM1 pays tothe other conference participants include how much (more specifically,analog values or stepped values indicating the state in which) theconference participant HM1 pays attention to each of the monitor devicesMD2 to MDn as well as whether (more specifically, digital “0” or “1”indicating whether) the conference participant HM1 pays attention in adirection toward each of the monitor devices MD2 to MDn or in anotherdirection.

The attention-degree-information generating section JB1 analyzes theimage data of the conference participant HM1, sent from the camera ofthe monitor device MDm to detect the direction in which the conferenceparticipant HM1 faces, every unit periods. Details thereof will bedescribed later.

More specifically, the attention-degree-information generating sectionJB1 detects a period Ij as shown in FIG. 3( a) as information indicatingthe direction in which the conference participant HM1 faces and a periodfor which the conference participant HM1 continues to face in thedirection. In this case, Ij is one of values 2 to n, which correspond tothe other conference participants HM2 to HMn, or 0, which indicates noneof the conference participants HM2 to HMn. Theattention-degree-information generating section JB1 detects a period 12in which the conference participant HM1 faces the monitor device MD2 onwhich an image of the conference participant HM2 is displayed, a periodI3 in which the conference participant HM1 faces the monitor device MD3on which an image of the conference participant HM3 is displayed, aperiod Im in which the conference participant HM1 faces the monitordevice MDm on which an image of the conference participant HMm isdisplayed, a period In−1 in which the conference participant HM1 facesthe monitor device MDn−1 on which an image of the conference participantHMn−1 is displayed, a period In in which the conference participant HM1faces the monitor device MDn on which an image of the conferenceparticipant HMn is displayed, or a period I0 in which the conferenceparticipant HM1 faces. none of the monitor devices MD2 to MDn.

Then, the attention-degree-information generating section JB1 detects aperiod longer than a time Tcont among detected periods, some of I2 to Inand I0. When the attention-degree-information generating section JB1detects a period longer than the time Tcont, it generates information(Hi:Aj) indicating that the conference participant HM1 pays attention tothe conference participant corresponding to the detected period, such asthose shown in FIG. 3( b).

In the information (Hi:Aj), “i” corresponds to a conference participantHMi (for example, “1” for HM1), and “j” corresponds to one of the otherconference participants HMj (i≠j) (2 to n corresponding to the otherconference participants HM2 to HMn when “i” is 1) or 0, whichcorresponds to none of the conference participants.

More specifically, a description is made in the case shown in FIG. 3.When the attention-degree-information generating section JB1 detects.periods I3, I0, and I2 as periods longer than the time Tcont amongdetected periods, some of I2 to In and I0, shown in FIG. 3( a), theattention-degree-information generating section JB1 generates, asattention-degree information, information (H1:A3) indicating that theconference participant HM1 pays attention to the conference participantHM3 corresponding to the detected period I3; information (H1:A0)indicating that the conference participant RM1 pays attention to none ofthe monitor devices MD or pay attention to something other than themonitor devices MD, corresponding to the detected period I0; andinformation (H1:A2) indicating that the conference participant HM1 paysattention to the conference participant HM2 corresponding to thedetected period 12, as shown in FIG. 3( b).

The attention-degree-information generating section JB1 may generateattention-degree information according to the detection of a period Ijand switch-pressing information sent from the switch SW. Morespecifically, when the attention-degree-information generating sectionJB1 detects periods Ij serving as information indicating the directionin which the conference participant HM1 faces and a period in which theconference participant HM1 faces in the direction, as shown in FIG. 4(a), and receives switch-pressing ON signals, such as those shown in FIG.4( b), obtained when the conference participant HM1 presses the switchSW during the detected periods, some of I2 to In and I0, theattention-degree-information generating section JB1 generatesinformation (H1:Aj), such as those shown in FIG. 4( c), indicating that,during a period when the switch-pressing signal is ON, the conferenceparticipant HM1 pays attention to the conference participantcorresponding to the signal. In the case shown in FIG. 4, when theattention-degree-information generating section JB1 detects 13 and 14 asperiods when the switch-pressing signal is ON, among the detectedperiods, some of I2 to In and I0, the attention-degree-informationgenerating section JB1 generates, as attention-degree information,information (H1:A3) indicating that the conference participant HM1 paysattention to the conference participant HM3 corresponding to thedetected period I3, and information (H1:A4) indicating that theconference participant HM1 pays attention to the conference participantHM4 corresponding to the detected period 14, as shown in FIG. 4( c).

In addition to the above cases, it is also possible that the conferenceparticipant HM1 explicitly specifies the direction in which theconference participant HM1 pays attention. For example, pushbuttonscorresponding to the other conference participants HM2 to HMn and apushbutton corresponding to a case in which the conference participantHM1 pays attention to none of those participants are prepared, and theconference participant HM1 specifies the direction in which theconference participant HM1 pays attention by pressing the correspondingpushbutton. In this case, pushbutton-pressing information serves as theattention-degree information.

3. Structure of Seating-Order Determination Device

Attention-degree information generated by theattention-degree-information generating section JB1 by determining whomthe conference participant HM1 pays attention among the conferenceparticipants HM2 to HMn according to a behavior or a designation of theconference participant HM1 as described above is transmitted to theinformation transmitting and receiving section TRB1 in the signalprocessing device SPD1, and then to a seating-order determination deviceGJD via the network connection terminal TN1 through the communicationnetwork NT.

The seating-order determination device GJD is structured as shown inFIG. 5.

In FIG. 5, the seating-order determination device GJD is provided with anetwork connection terminal 72 for connecting to the communicationnetwork NT; an information transmitting and receiving section 70 fortransmitting and receiving information to and from the communicationnetwork NT; and a seating-order determiner 71 for determining a seatingorder according to attention-degree information sent from theteleconference devices TCD1 to TCDn, for generating seating-orderinformation indicating the seating order, and for sending theseating-order information to the information manipulation anddistribution sections PB of the teleconference devices TCD1 to TCDn.

More specifically, in the seating-order determination device GJD, theinformation transmitting and receiving section 70 picks up theattention-degree information sent from the teleconference devices TCD1to TCDn among signals passing through the communication network NT, andsends the attention-degree information to the seating-order determiner71. The seating-order determiner 71 determines the seating order of theconference participants HM1 to HMn attending the conference through theteleconference devices TCD1 to TCDn, and generates seating-orderinformation indicating the determined seating order.

The information transmitting and receiving section 70 transmits thegenerated seating-order information to the communication network NT tosend it to the teleconference devices TCD1 to TCDn.

Although details will be described later, the teleconference devicesTCD1 to TCDn receive the seating-order information sent from theseating-order determination device GJD at the information manipulationand distribution sections PB; and the information manipulation anddistribution sections PB determine the correspondences betweenconference participants HM and monitor devices MD according to theseating-order information as described later, applies video and audiomanipulation such that, for example, a change in seating order is easyto understand, and sends images and sound related to the conferenceparticipants HM corresponding to the monitor devices MD2 to MDn toimplement the determined seating order.

4. Grouping Processing in the Seating-Order Determination Device

Various methods for determining a seating order can be considered forthe seating-order determination device GJD. A method can be consideredas an example, in which a conversation group to which conferenceparticipants HM1 to HMn belong is determined and then a seating order isdetermined according to the result of group determination such thatparticipants belonging to the same group are set close to each other.

A process for determining a conversation group in the seating-orderdetermination device GJD will be described below before seating-orderdetermination processing performed according to a group is described.

Various group-determination rules used in the seating-orderdetermination device GJD can be considered. In the followingdescription, a rule in which a link is made between a person who paysattention to another person and the another person, who attractsattention, and one group is formed of persons who are coupled directlyor indirectly by links is used as an example. Details of a groupdetermination and update process performed by the rule according toattention-degree information in the seating-order determiner 71 of theseating-order determination device GJD will be described below. In thisrule, one group is formed of persons who directly or indirectly payattention to others, and the others, who attract attention directly orindirectly.

A condition in which a person directly or indirectly pays attention toanother person, and the another person attracts attention directly orindirectly will be described with the following example. It is assumed,as an example, that a person A pays attention to a person B, and theperson B pays attention to a person C. In this example, it is consideredthat the person A pays attention to the person B “directly” (the personB also pays attention to the person C “directly”) and the person Battracts attention “directly;” and the person A pays attention to theperson C “indirectly (through the person B)” and the person C attractsthe attention of the person A “indirectly (through the person B).” Inthis example, there is only one person, the person B, between the personA and the person C. But, there may be a plurality of persons between theperson A and the person C.

The seating-order determiner 71 holds an attention-destination tableformed of an “individual number” column indicating numbers assigning toconference participants attending a conference held with thisteleconference system; an “attention-destination number” columnindicating the numbers of conference participants whom the conferenceparticipants HM1 to HMn using the teleconference devices TCD1 to TCDnpay attention; and a “whether registration has been made to group table”column indicating whether registration has been made to a group table,as shown in FIG. 6.

The seating-order determiner 71 also holds a group table formed of a“group number” column indicating numbers assigned to groups formed inthe teleconference system; a “number of members” column indicating thenumber of members belonging to each group among the conferenceparticipants; and a “member” column indicating conference participantsbelonging to the groups, as shown in FIG. 7.

The seating-order determiner 71 performs initialization at the start ofcommunication prior to receiving attention-degree information, as shownin FIG. 8.

More specifically, the seating-order determiner 71 generates theattention-destination table shown in FIG. 6 and performs initializationin the process of step S31 shown in FIG. 8.

In the initialization, the seating-order determiner 71 sets the“individual number” column to the numbers (H1 to Hn in a case shown inFIG. 6) corresponding to conference participants attending a conferenceheld with the teleconference system in the attention-destination tableshown in FIG. 6; sets the “attention-destination number” column all to anumber A0 which indicates that the conference participants HM1 to HMnwho use the teleconference devices TCD1 to TCDn pay attention to none ofthem; and sets the “whether registration has been made to group table”column to all “x” which indicates that registration has not yet beenmade to any group.

The seating-order determiner 71 generates a group table and performsinitialization in the process of step S32 as shown in FIG. 7.

In the initialization, the seating-order determiner 71 sets the “groupnumber” column to G1 to Gn which indicate that each of theteleconference devices TCD1 to TCDn forms one group, in the group tableshown in FIG. 7; sets the “number of members” column to all 0, whichindicates that none attends any group; and sets the “member” column toall null, which indicates that any group has no member.

When the seating-order determiner 71 receives attention-degreeinformation, it starts a process to be performed when attention-degreeinformation is generated, as shown in FIG. 9.

In FIG. 9, when the seating-order determiner 71 receivesattention-degree information, it sets the attention-destination tableaccording to the attention-degree information in the process of step S41as shown in FIG. 10.

Actually, the seating-order determiner 71 specifies the“attention-destination number” column such that, when information(Hi:Aj) is received, which indicates that the conference participant HMiindicated by Hi pays attention to the conference participant HMjcorresponding to Aj, the “attention-destination number” cellcorresponding to an individual number of Hi is set to Aj.

The “whether registration has been made to group table” column is allinitialized to “x.”

In a case shown in FIG. 10, the “attention-destination number” cellcorresponding to an “individual number” of H1 is set to A3 correspondingto the conference participant HM3 as the attention destination of theconference participant HM1 indicated by an “individual number” of H1;the “attention-destination number” cell corresponding to an “individualnumber” of H2 is set to A0, which indicates that none attractsattention, as the attention destination of the conference participantHM2 indicated by an “individual number” of H2; the“attention-destination number” cell corresponding to an “individualnumber” of H3 is set to A5 corresponding to the conference participantHM3 as the attention destination of the conference participant HM3indicated by an “individual number” of H3; and the“attention-destination number” cell corresponding to an “individualnumber” of H4 is set to A2 corresponding to the conference participantHM2 as the attention destination of the conference participant HM4indicated by an “individual number” of H4. A description for conferenceparticipants HM5 to HMn will be omitted. The “attention-destinationnumbers” column is specified for HMi in this way.

The seating-order determiner 71 sets the “numbers of members” column allto 0 and the “member” column all to null as shown in FIG. 7 in the grouptable in the process of step S42.

After steps S42 and S43, the seating-order determiner 71 sets the grouptable according to the contents of the attention-destination table foreach of “individual numbers” H1 to Hn in the process of step S43.

FIG. 11 is a detailed flowchart of processing for setting the grouptable according to the contents of the attention-destination table instep S43 of the flowchart of FIG. 9. In FIG. 11, “registration” meansregistration to the group table; a “self entry” means a self entry inthe attention-destination table; an “attention destination” means anattention destination in a self entry; an “attention-destination entry”means the entry of an attention destination in the attention-destinationtable; a “self group” means a group to which the self belongs as amember; and an “attention-destination group” means a group to which anattention destination belongs as a member. An actual registrationoperation means that the number of members is incremented by one in agroup table; a corresponding number is added as a member in the grouptable; and the corresponding “whether registration has been made togroup table” cell in the attention-destination table is changed to “◯.”

A requirement concerning the setting of the entry of a conferenceparticipant HMi in the attention-destination table will be firstdescribed. A requirement (hereinafter called a first requirement in eachteleconference device) concerning the setting of the entry of theconference participant HMi having an individual number of Hi in theattention-destination table includes the registration of the individualnumber Hi to the group table, and a change of the “whether registrationhas been made to group table” cell in the entry of the individual numberHi in the attention-destination table to “◯.” When theattention-destination number corresponding to the individual number Hiis A0 (which means the conference participant pays attention to none ofthe other conference participants), only this requirement applies.

When the attention-destination number corresponding to the individualnumber Hi is not A0, a requirement (hereinafter called a secondrequirement in each teleconference device) includes the registration ofthe attention-destination number to the group table, a change of the“whether registration has been made to group table” cell in the entry ofthe attention destination in the attention-destination table to “◯,” andthe registration of the individual number Hi and the attentiondestination to the same group in the group table.

By following the flowchart shown in FIG. 11, whether these requirementsare satisfied is checked.

In the process of step S51, the seating-order determiner 71 determineswhether the “whether registration has been made to group table” cell inthe self entry of the individual number Hi in the attention-destinationtable is “◯.” When the “registration” in the self entry has already beenset to “◯” in step S51, since it is ensured that registration to thegroup table has already been performed, the first requirement is alreadysatisfied. Therefore, when it is determined in step S51 that YES isobtained, the seating-order determiner 71 does not perform work relatedto registration of the individual number Hi to the group table, and theprocessing proceeds to step S53. When it is determined in step S51 thatNO is obtained, since the “registration” is “x” in the self entry, theprocessing proceeds to step S52.

In the process of step S52, the seating-order determiner 71 registersthe self (individual number) to a group having no member and thesmallest number in the group table, and the processing proceeds to stepS53.

As the results of steps S51 and S52, the first requirement is satisfiedbefore the process of the next step S53.

In the process of step S53, the seating-order determiner 71 determineswhether the attention-destination number is not A0. When theattention-destination number is A0, in other words, it is determined instep S53 that NO is obtained, since all requirements have already beensatisfied, the seating-order determiner 71 finished the processing. Whenthe attention-destination number is not A0, in other words, when it isdetermined in step S53 that YES is obtained, the processing proceeds tostep S54 to satisfy the second requirement.

In the process of step S54, the seating-order determiner 71 determineswhether the “whether registration has been made to group table” cell inthe attention-destination entry is “◯.”

When the “registration” is “x” in the attention-destination entry, inother words, it is determined in step S54 that NO is obtained, since itis sure that the attention-destination entry has not been registered tothe group table, the processing proceeds to step S58, and theseating-order determiner 71 registers the attention destination to thegroup (self group) to which the self belongs, and sets the “whetherregistration has been made to group table” cell in theattention-destination entry in the attention-destination table to “◯.”Since the second requirement is surely satisfied with the process ofstep S58, the seating-order determiner 71 finishes the processing.

When it is determined in step S54 that the “registration” is “◯” in theattention-destination entry, which means that the attention destinationhas already been registered, only the last item of the secondrequirement needs to be satisfied, which indicates that the individualnumber Hi and the attention destination belong to the same group. Whenit is determined in step S54 that YES is obtained, the seating-orderdeterminer 71 refers to the group table in the next step S55 and checkswhether the individual number Hi and the attention destination belong tothe same group in step S56.

When it is confirmed in step S56 that they belong to the same group, inother words, when it is determined in step S56 that YES is obtained,since the second requirement has already been satisfied, theseating-order determiner 71 finishes the processing.

When it is determined in step S56 that No is obtained, since theindividual number Hi and the attention destination need to belong to thesame group, the seating-order determiner 71 merges the two groups towhich the individual number Hi and the attention destination belong inthe process of step S57. More specifically, the seating-order determiner71 merges (adds the number of members in the group having a largernumber to that in the group having a smaller number, adds the members ofthe group having the larger number to those of the group having thesmaller number, sets the number of members in the group having thelarger number to zero, and sets the members of the group having thelarger number to null) the group having a larger number into the grouphaving a smaller number among the two groups. As a result, the secondrequirement is satisfied. Since all the requirements have beensatisfied, the seating-order determiner 71 finishes the processing.

The confirmation of the requirements related to the settings of theentry of the conference participant HMi in the attention-destinationtable has been finished.

When the entry corresponding to the individual number Hi has been set inthe attention-destination table, it is necessary that both the entrycorresponding to the individual number Hi and those corresponding to theindividual numbers up to Hi−1 be set in the attention-destination table.As for the setting of the entry corresponding to the individual numberH1, since the other entries have not yet set so far, it is required thatonly the entry corresponding to the individual number H1 be set.

Operations which may be performed when the entry corresponding to theindividual number Hi is input include a new registration to the grouptable, the merger of two groups in the group table, and a change in the“whether registration has been made to group table” cell in theattention-destination table to “◯.” With these operations, neithernumbers registered according to the entries corresponding to individualnumbers up to Hi−1, followed by Hi, are deleted, nor the “whetherregistration has been made to group table” cells are set to “x.” Inaddition, two numbers belonging to the same group are not separated,either. Therefore, the process for setting the entry corresponding tothe individual number Hi into the tables does not break the requirementssatisfied by the processes for the individual numbers up to Hi−1,followed by Hi. Consequently, it is inductively understood that, whenthe processes related to the entries of all individual numbers Hi, shownin FIG. 11, have been finished, the requirements for the entries of allindividual numbers H1 to Hi are satisfied.

Changes in the attention-destination table and in the group table, madein the processing shown by the flowchart of FIG. 11 will be describedbelow by examples. It is assumed here that the group table has a stateshown in FIG. 7 as initial states, and the attention-destination tablehas a state shown in FIG. 10 when group-determination information hasbeen input. Changes in the attention-destination table and the grouptable will be explained.

The setting of the entry of the conference participant HM1 (having anindividual number of H1) will be described first.

Since the “whether registration has been made to group table” cell is“x,” it is determined in step S51 of FIG. 11 that NO is obtained, andthe entry corresponding to the individual number Hi is input to thegroup table in the next step S52. As a result of the process of stepS52, the “number of members” for a group number G1 having the smallestnumber is set to one, and the individual number H1 is input to the“member” column, as shown in FIG. 12. The “whether registration has beenmade to group table” cell for the individual number H1 is changed to “◯”in the attention-destination table as shown in FIG. 13.

Then, it is determined in step S53 that YES is obtained. Since theattention destination is A3 indicating the conference participant HM3,and the “whether registration has been made to group table” cell for theentry of the individual number H3 is “x,” as shown in FIG. 13, it isdetermined in step S54 that NO is obtained. As a result, in step S58,the individual number H3 of the attention destination is input to thegroup G1, which is the “self group.” In the group table, the “number ofmembers” in the group G1 is set to two and its members are H1 and H3 inthe group table as shown in FIG. 14. In the attention-destination table,the “whether registration has been made to group table” cells for theindividual numbers H1 and H3 are set to “◯” as shown in FIG. 15.

The setting of the entry of the conference participant HM1 (having anindividual number of Hi) has been finished.

The setting of the entry of the conference participant HM2 (having anindividual number of H2) will be described next.

Since the “whether registration has been made to group table” cell is“x” when the entry corresponding to the individual number H2 is input tothe tables, it is determined in step S51 of FIG. 11 that NO is obtained,and the entry corresponding to the individual number H2 is input to thegroup table in the next step S52. As a result of the process of stepS52, the number of members for a group number G2 having the nextsmallest number is set to one, and the individual number H2 is input tothe member column, as shown in FIG. 16. The “whether registration hasbeen made to group table” cell for the individual number H2 is changedto “◯” in the attention-destination table as shown in FIG. 17. Since theattention-destination number of the individual number H2 is A0 as shownin the case of FIG. 17, the setting of the entry of the conferenceparticipant HM2 has been finished at this point.

The setting of the entry of the conference participant HM3 (having anindividual number of H3) will be described next.

Since the “whether registration has been made to group table” cell is“◯” as shown in FIG. 15 when the entry corresponding to the individualnumber H3 is input to the tables, because the entry corresponding to theindividual number H1 has been set. Therefore, it is determined in stepS51 of FIG. 11 that YES is obtained. Since the conference participantHM3 having the individual number H3 pays attention to the conferenceparticipant HM5, it is determined in step S53 that YES is obtained.Since the “whether registration has been made to group table” cell forthe individual number H5, indicating the attention destination, is “x,”as shown in FIG. 17, it is determined in step S54 of FIG. 11 that NO isobtained.

As a result, in step S58, the individual number H5 of the attentiondestination is input to the group G1, which is the “self group” of theindividual number H3. In the group table, the number of members in thegroup G1 is updated to three and its members are updated to H1, H3, andH5 in the group table as shown in FIG. 18. In the attention-destinationtable, the “whether registration has been made to group table” cell forthe individual number H5 is set to “◯” as shown in FIG. 19.

The setting of the entry of the conference participant HM3 (having anindividual number of H3) has been finished.

The setting of the entry of the conference participant HM4 (having anindividual number of H4) will be described next.

Since the “whether registration has been made to group table” cell is“x” when the entry corresponding to the individual number H4 is input tothe tables, it is determined in step S51 of FIG. 11 that NO is obtained,and the entry corresponding to the individual number H4 is input to thegroup table in the next step S52. As a result of the process of stepS52, the number of members for a group number G3 which is the groupfollowing the groups G1 and G2 is set to one, and the individual numberH4 is input to the member column, as shown in FIG. 20. The “whetherregistration has been made to group table” cell for the individualnumber H4 is changed to “◯” in the attention-destination table as shownin FIG. 21.

Then, it is determined in step S53 that YES is obtained. Since theattention destination is A2 indicating the conference participant HM2,and the “whether registration has been made to group table” cell for theentry of the individual number H2 is “◯,” as shown in FIG. 21, it isdetermined in step S54 that YES is obtained.

In step S55, a group number is searched for the individual number H2 ofthe attention destination to obtain the group number G2 to which theindividual number H2 serving as the attention destination belongs. Inthe next step S56, it is determined that NO is obtained because thegroup number to which the individual number H4 belongs is G3 and thegroup number to which the individual number H2 serving as the attentiondestination belongs is G2. In the next step S57, the group number G3 towhich the individual number H4 belongs is merged into the group G2,which has a smaller number, to which the individual number H2 belongs.

Therefore, in the group table, the number of members in the group G2 isupdated to two, and its members are updated to H2 and H4, as shown inFIG. 22. The attention-destination table is updated (the same as thatshown in FIG. 21) as shown in FIG. 23.

The setting of the entry of the conference participant HM4 (having anindividual number of H4) has been finished.

The setting of the entry of the conference participant HM5 (having anindividual number of H5) will be described last.

Since the “whether registration has been made to group table” cell is“◯” as shown in FIG. 23 when the entry corresponding to the individualnumber H5 is input to the tables. Therefore, it is determined in stepS51 of FIG. 11 that YES is obtained. Since the conference participantHM5 having the individual number H5 pays attention to the conferenceparticipant HM3, it is determined in step S53 that YES is obtained.Since the “whether registration has been made to group table” cell forthe individual number H3, indicating the attention destination, is “◯”as shown in FIG. 23, it is determined in step S54 of FIG. 11 that YES isobtained.

In step S55, a group number is searched for the individual number H3 ofthe attention destination to obtain the group number G1 to which theindividual number H3 serving as the attention destination belongs. Inthe next step S56, it is determined that YES is obtained because thegroup number to which the individual number H3 serving as the attentiondestination belongs is G1, and the individual number H5 has already beenregistered to the group G1. The processing is finished.

The group table is updated (is the same as that shown in FIG. 22) asshown in FIG. 24, and the attention-destination table is updated (is thesame as that shown in FIG. 23) as shown in FIG. 25.

With the processing described above, when information related to allindividual numbers are input to the attention-destination table, the“registration to group table” column is set to all “◯” in theattention-destination table. All individual numbers have already beenregistered to any one of groups as members in the group table.

5. Seating-Order Determination Operation Through Grouping in theSeating-Order Determination Device

When group determination is finished in the processing described above,the seating-order determination device GJD determines a seating ordersuch that persons belonging to the same group are collectively arranged.

To this end, the seating-order determination device GJD holdsseating-order information as well as group information.

When a group is changed, the seating-order determination device GJDrefers to the held seating-order information and changes it byprocessing shown in a flowchart of FIG. 26 to determine a new seatingorder. This processing will be described below.

In step S101, it is determined whether there is a group (hereinaftercalled a divided group) in which its members are separated.

When no group is divided at the seating order used before a change, inother words, when members are collectively arranged in all groups, sincethe target condition is satisfied, the seating order used before achange is used as is. The processing shown in FIG. 26 is finished.

When it is determined in step S101 that there is a divided group, agroup having the largest number of members is determined among dividedgroups in step S102. When there are a plurality of divided groups, agroup having the minimum group number is regarded as the largest group.Since the largest group is divided, if a set of one person or morecollectively arranged is called a sub group, the largest group is formedof a plurality of sub groups.

When the largest divided group is determined, the largest sub group isdetermined in the largest divided group and a sub group located closestto the largest sub group is determined in step S103. When there are aplurality of groups having the largest number of members among subgroups, a group to which a person having the minimum individual numberbelongs is regarded, for example, as the largest sub group. Whenseparate sub groups are located at the same distance from the largestsub group clockwise and counterclockwise, the sub group located at thedistance counterclockwise is regarded, for example, as the sub groupclosest to the largest group.

When the sub group closest to the largest sub group is determined, thedetermined sub group is connected to the largest sub group in step S104.This connection process will be described below.

The determined sub group closest to the largest sub groupcounterclockwise from the largest sub group is shifted clockwise toconnect to the largest sub group. The determined sub group closest tothe largest sub group clockwise from the largest sub group is shiftedcounterclockwise to connect to the largest sub group. When thedetermined sub group is located at the same distance from the largestsub group clockwise and counterclockwise, the determined sub group is,for example, shifted clockwise to connect to the largest sub group.Persons located between the largest sub group and the sub group closestthereto are shifted by the number of members belonging to the sub groupclosest to the largest sub group in the direction opposite that in whichthe sub group closest to the largest sub group is shifted.

Details of the connection process have been described.

Since the number of sub groups in the largest divided group is reducedby one by the above-described connection process, the largest dividedgroup is collectively arranged by the repetition of the process shown inthe flowchart of FIG. 26.

When the process for one divided group is finished, the same process canbe applied to the next largest group by the repetition of the processshown in the flowchart. Therefore, each of all groups is collectivelyarranged by the repetitions of the process shown in the flowchart, andthe seating-order determination processing is finished with the targetcondition being obtained.

Two example seating-order changes in the seating-order determinationprocessing will be described below.

FIG. 27 shows a first example. In a state shown in FIG. 27( a), only agroup G1 indicated by black circles is divided into sub groups SG1 andSG2. The sub group SG2 closest to the largest sub group SG1 is locatedin the counterclockwise direction.

In this case, the connection process performs shifting indicated byarrows in FIG. 27( a) to change a seating order to that shown in FIG.27( b), and the seating-order determination processing is completed.

FIG. 28 shows a second example. In a state shown in FIG. 28( a), twogroups, a group G1 indicated by black circles and a group G2 indicatedby white circles, are divided. The largest divided group is the groupG1. The group G1 is divided into three sub groups SG1, SG2, and SG3. Thesub groups SG2 and SG3 are located at the same distance from the largestsub group SG1 clockwise and counterclockwise. In this case, according tothe above-described rule, shifting is performed first so as to connectthe sub group SG3, located in the counterclockwise direction from thelargest sub group.

Then, as shown in FIG. 28( b), two sub groups SG11 and SG2 are disposed.The sub group closest to the largest sub group SG11 is SG2, and locatedin the clockwise direction from the largest sub group SG11.

The connection process is applied to the sub group SG2 to obtain a stateshown in FIG. 28( c), in which the group G1 is collectively arranged.

As a result of the process, another divided group G2 is collectivelyarranged. A total to two repetitions of the connection process completesthe seating-order determination processing.

When the seating-order determination processing is completed asdescribed above, for example, the seating-order determiner 71 generatesseating-order information showing the determined seating order and sendsit to each teleconference device TCD. Together with the seating-orderinformation, the seating-order determiner 71 sends group informationindicating the state of grouping.

In the above-described case, the seating-order determiner 71 determinesthe seating order according to the result of group determination suchthat conference participants belonging to the same groups arecollectively arranged. To make a viewing range small when watchingmembers belonging to the same group, members belonging to the samegroups may be arranged so as to be uniformly dispersed.

As shown in FIG. 29, for example, a seating order may be determined suchthat conference participants belonging to groups G1 and G2 are uniformlydispersed.

6. Seating-Order Changing Processing Performed According toSeating-Order Information in a Teleconference Device

Operations of a teleconference device TCD, performed when image data andaudio data sent from each teleconference device TCD, and theabove-described seating-order information sent from the seating-orderdetermination device GJD are received in the teleconference systemaccording to the present embodiment will be described below. Theteleconference device TCD1 will be taken as an example among theteleconference devices TCD1 to TCDn and its operations will bedescribed.

When the information transmitting and receiving section TRB1 of theteleconference device TCD1 receives a signal sent through thecommunication network NT, the information transmitting and receivingsection TRB1 separates the image data and the audio data correspondingto the teleconference devices TCD2 to TCDn from the signal; picks up theabove-described seating-order information (including the groupinformation); and sends the picked-up seating-order information as wellas the separated image data and audio data to the informationmanipulation and distribution section PB1.

The information manipulation and distribution section PB1 distributesinput images and/or sound to the corresponding monitor devices MDaccording to the seating-order information. The information manipulationand distribution section PB1 may manipulate images and/or sound to bedistributed, so that, for example, a seating-order change is made easyto understand intuitively.

A specific process to be performed in the information manipulation anddistribution section PB1 will be described below by taking a case as anexample, in which manipulation is applied to images and sound so as tomake a seating-order change easy to understand intuitively, and imagesand sound are sent to the corresponding monitor devices MD according tothe seating-order information.

FIG. 30 shows the structure of the information manipulation anddistribution section PB1. It includes an input terminal 201 forreceiving images and sound received by the information transmitting andreceiving section TRB1 shown in FIG. 2; an input terminal 202 forreceiving seating-order information received by the informationtransmitting and receiving section TRB1; a motion determination section203 for performing motion determination related to the seating-orderinformation; a connection determination section 204 for determining aconnection state according to the seating-order information; an imagemanipulation device 205 and an audio manipulation device 206 formanipulating input images and sound; and an information distributionsection 207 for distributing manipulated images and sound to eachmonitor device MD.

The output terminals TO2 to TOn connected to the informationdistribution section 207 are those used for sending images and sound tothe monitor devices MD2 to MDn as shown in FIG. 2.

The motion determination section 203 determines the directions andamounts of the relative motions of conference participants HM located atremote places against the conference participant HM1 located on siteaccording to the input seating-order information, namely, determines thedirections and amounts of motions related to changes in a seating order,and sends the results as motion information to the image manipulationsection 205 and to the audio manipulation section 206.

FIG. 31 shows the structure of the image manipulation device 205. Theimage manipulation device 205 includes image manipulators 250-2, 250-3,. . . , and 250-n for manipulating the input images corresponding toconference participants HM.

The images of conference participants HM are sent from an input terminal251 to the image manipulators 250-2, 250-3, . . . , and 250-n, and themotion information is sent from the motion determination section 203through an input terminal 252 to the image manipulators 250-2, 250-3,and 250-n.

Each of the image manipulators 250-2, 250-3, . . . , and 250-n extractsa motion related to the corresponding conference participant HM from themotion information, and manipulates the input image of the correspondingconference participant HM that the movement of the conferenceparticipant HM is intuitively easy to understand, as required, accordingto the extracted motion.

The manipulated images are output from an output terminal 253 to theinformation distribution section 207.

In an example manipulation performed in each of the image manipulators250-2, 250-3, . . . , and 250-n, arrows indicating the directions of therelative motions of conference participants HM located at remote placesagainst the conference participant HM1 located on site are superposed oninput images.

FIG. 34 (a) shows an input image, and FIG. 34( b) shows a manipulatedimage obtained when a motion direction is left.

When a relative motion is not found, a method in which an arrow is notsuperposed may be used. An arrow is superposed until the timeimmediately before a connection is changed by the informationdistribution section 207, described later.

In another example manipulation performed in each of the imagemanipulators 250-2, 250-3, . . . , and 250-n, input images are moved onscreens in the directions of the relative motions of conferenceparticipants HM located at remote places against the conferenceparticipant HM1 located on site.

FIG. 35( a) shows an input image, and FIG. 35( b), (c), and (d) showsmanipulated images obtained every time when a predetermined timeelapses, if the motion direction is left. Input images are moved untilthe time immediately before a connection is changed by the informationdistribution section 207.

In still another example manipulation performed in each of the imagemanipulators 250-2, 250-3, . . . , and 250-n, input images are moved onscreens as described above with the background being fixed and only theportion of the conference participants HM in the input images beingmoved.

To implement the above-described manipulation, there is a method inwhich backgrounds viewed from the cameras corresponding to conferenceparticipants HM located at remote places are set to blue backgrounds(BB); portions other than the blue backgrounds are extracted asconference participants from input images by the image manipulators250-2, 250-3, . . . , and 250-n; the portions are shifted according tothe corresponding motions and then attached back to the images; and afixed background is attached to the parts other than the portionsattached in the images.

FIG. 36( a) shows an example input image, obtained before movement, FIG.36( b) shows a manipulated image obtained when a predetermined timeelapses, and FIG. 36( c) shows a manipulated image obtained when apredetermined time further elapses, if a motion direction is left. Asshown in the figure, the image of the conference participant is shifted,for example, in the left direction according to the direction in which aseating order is changed with a blue background (BB) being used as abackground. The image is shifted according to the corresponding motion,for example, until the time immediately before a connection is changedby the information distribution section 207, described later.

When the seating order of conference participants displayed on themonitor devices MD2 to MDn is changed, these image manipulationprocesses make the change and a direction in which the change isperformed easy to understand for the conference participant HM1.

FIG. 32 shows the structure of the audio manipulation device 206. Theaudio manipulation device 206 includes audio manipulators 260-2, 260-3,. . . , and 260-n for manipulating the input sound corresponding toconference participants HM.

The sound of conference participants HM are sent from an input terminal261 to the audio manipulators 260-2, 260-3, . . . , and 260-n, and themotion information is sent from the motion determination section 203through an input terminal 262 to the image manipulators 260-2, 260-3,and 260-n.

Each of the audio manipulators 260-2, 260-3, . . . , and 260-n extractsa motion related to the corresponding conference participant HM from themotion information, and manipulates the input sound of the correspondingconference participant HM so that the movement of the conferenceparticipant HM is intuitively easy to understand, as required, accordingto the extracted motion.

The manipulated sound is output from an output terminal 263 to theinformation distribution section 207.

In an example manipulation performed in each of the audio manipulators260-2, 260-3, . . . , and 260-n, a message such as “the seating order isbeing changed” and/or sound indicating a change of the seating order issuperposed on sound when a conference participant HM located at a remoteplace relatively moves against the conference participant HM1 located onsite. The message and/or sound is superposed, for example, until thetime immediately before or immediately after a connection is changed bythe information distribution section 207, described later.

In another example manipulation, a message such as “the seating order isbeing changed” and/or sound indicating a change of the seating order issuperposed on sound irrespective of the motion of a conferenceparticipant HM located at a remote place. The message and/or sound issuperposed, for example, until the time immediately before orimmediately after a connection is changed by the informationdistribution section 207, described later.

The connection determination section 204, shown in FIG. 30, determines amethod for connecting images and sound to each monitor device MDaccording to the seating order information input from the input terminal202, and sends it to the information distribution section 207 asconnection information.

FIG. 33 shows the structure of the information distribution section 207.Images sent from the image manipulation device 205 through an inputterminal 271, and sound input from the sound manipulation device 206through an input terminal 272 are sent to a matrix switcher 270. Theimages and sound are sent to each monitor device MD so as to conform tothe seating-order information, according to the connection informationsent from the connection determination section 204 through an inputterminal 273 to the matrix switcher 270. In other words, the matrixswitcher 270 switches the images and sound to the output terminals TO2to TOn according to the connection information.

Since the information manipulation and distribution section PB has theabove-described structure in the signal processing device SPD of eachteleconference device TCD, a dynamic seating-order change is performedin the monitor devices MD2 to MDn according to the seating-orderinformation generated by the seating-order determination device GJD.

The relationships (seating order) between the monitor devices MD2 to MDnand conference participants HM2 to HMn shown thereon are flexiblychanged according to conversation groups made and released during aconference such that a suitable condition is made for the conferenceparticipant HM1 to do conversation.

In the foregoing description, both images and sound are handled asinformation related to conference participants HM located at remoteplaces. One of them may be handled as the information.

In the foregoing description, images and sound are manipulated. Inputimages and/or sound may be directly sent to the information distributionsection 207 without being manipulated.

When a seating-order is frequently changed, a confusion may occur as towhich conversation each conference participant is doing.

To avoid such a condition, when the seating-order determination deviceGJD determines a seating order according to group determination asdescribed above, a process for making the background of each conferenceparticipant HM in the corresponding image have similarity in units ofgroups can be applied in image processing performed by the informationmanipulation and distribution section PB.

More specifically, for example, backgrounds viewed from the camerascorresponding to conference participants HM located at remote places areset to blue backgrounds; the information manipulation and distributionsection PB extracts these blue backgrounds as backgrounds; and changesthem to backgrounds having the same colors in units of groups. Theinformation manipulation and distribution section PB sets the backgroundcolor of each image and performs image processing according to the groupinformation. A method for setting the backgrounds of conferenceparticipants who belong to the same group as the self (conferenceparticipant HM1) to, for example, a fixed color (such as blue) can beconsidered so as to understand the conference participants belonging tothe same group as the conference participant HM1.

FIG. 37A and FIG. 37B show example background colors used before andafter conversion. In the figure, G1 to G3 indicate the numbers of groupsto which conference participants belong. In this case, in a stateobtained after the conversion shown in FIG. 37B, the conferenceparticipants belonging to the group G2 understand that the conferenceparticipant having the blue background belongs to the same group, andeasily understands that the group G1 having a red background and thegroup G3 having a green background are formed.

7. First Example of Grouping Processing which Uses a StatisticalRelationship, in the Seating-Order Determination Device

In the foregoing case, the seating-order determination device GJD uses agroup determination method based on the rule in which a link is madebetween a person who pays attention to another person and the anotherperson, who attracts attention, and one group is formed of persons whoare coupled directly or indirectly by links. In another case, a groupcan be determined by statistical relationships between attentionpatterns which indicate combinations of the attention destinations ofconference participants, and group patterns.

In such a case, a process can be used in which the seating-orderdetermination device GJD prepares a group determination table like thatshown in FIG. 40, and converts an attention pattern to a group patternaccording to the table.

A method for preparing such a group determination table in advanceaccording to statistics will be described below. A case in which thenumber of conference participants is three will be taken as an example.

To make a group determination table, an experiment is performed to haveactual conversation states, and time-sequential samples formed ofcombinations of attention patterns and group patterns are prepared, forexample, at a predetermined interval.

It is possible, for example, that attention patterns are automaticallyobtained and group patterns are determined by a person.

Two methods for generating a group determination table according to thesamples can be considered. A first method will be described first.

In this method, the most frequently generated group pattern is found foreach attention pattern, and the group pattern is registered as the grouppattern corresponding to the attention pattern.

An experiment is first performed in this method to record a pattern ofthe attention destinations of conference participants and a grouppattern formed at that time at a predetermined interval to generate afrequency table between attention patterns and group patterns, like thatshown in FIG. 39.

In FIG. 39, numbers (zero indicates that a conference participant paysattention to nobody) indicated below conference participants HM1 to HM3are those of the attention destinations of the conference participants,and GP1 to GP5 indicate the numbers of group patterns.

FIG. 38 shows example definitions of group patterns.

Various group forms are defined as group patterns GP1 to GP5 as shown inFIG. 38, in which a group pattern GP1 indicates that three conferenceparticipants HM1 to HM3 have no group, a group pattern GP2 indicates astate in which conference participants HM1 and HM2 form a group, . . . .

The frequency table shown in FIG. 39 indicates that, as a result of theexperiment, when the attention destinations of the conferenceparticipants HM1 to HM3 are all zero, the group pattern GP1 is formed10034 times, the group pattern GP2 is formed 130 times, . . . , and thegroup pattern GP5 is formed 3024 times.

A group pattern is determined in the experiment, for example, by aperson who sees the state.

According to such a frequency table, the group pattern corresponding toan attention pattern of the conference participants HM1 to HM3 isselected so as to have the highest probability, and the groupdetermination table shown in FIG. 40 is made from the selectedcorrespondences.

In the frequency table shown in FIG. 39, when the attention destinationsof the conference participants HM1 to HM3 are all zero, the grouppattern GP1 has the highest frequency. Therefore, when the conferenceparticipants HM1 to HM3 have an attention destination of 0, which isshown by (0, 0, 0), the group pattern is set to GP1.

When the conference participant HM3 has an attention destination of 1and the other conference participants HM1 and HM2 have an attentiondestination of 0, which is shown by (0, 0, 1), the group pattern is setto GP4 according to the frequency table shown in FIG. 39.

A group pattern having the highest frequency is determined for each ofall attention-destination patterns to make the relationships betweenattention-destination patterns and group patterns shown in FIG. 40.

The seating-order determination device GJD holds a group determinationtable generated in advance, like that described above, to determine thegroup corresponding to an attention pattern of the conferenceparticipants HM1 to HM3 by referring to the group determination tablewhen it receives attention-degree information from the teleconferencedevices TCD, and generates seating-order information according to thegroup determination.

8. Second Example of Grouping Processing which Uses a StatisticalRelationship, in the Seating-Order Determination Device

In the foregoing case, all attention patterns are independently handledwhen the frequency table between attention patterns and group patters isgenerated. In a second case, attention patterns similar to each otherare collectively handled to generate a frequency table, and a groupdetermination table is formed according the frequency table.

When attention-destination patterns of the conference participants HM1to HM3 are (0, 0, 2), (0, 1, 0), (0, 3, 0), (2, 0, 0), and (3, 0, 0),for example, their mutual attention-destination relationships aresubstantially the same as attention destinations obtained by rotatingand/or inverting those of (0, 0, 1). Therefore, for example, the grouppatterns GP1, GP2, GP3, GP4, and GP5 corresponding to an attentionpattern of (0, 0, 2) are regarded as the same as the group patterns GP1,GP2, GP3, GP4, and GP5 corresponding to an attention pattern of (0, 0,1), obtained by inverting (applying inversion for a segment drawn fromthe conference participant HM3 to the middle of the conferenceparticipants HM1 and HM2 to) the group patterns of the attention patternof (0, 0, 2), similar attention patterns are collectively handled as anattention pattern of (0, 0, 1), and then statistics is obtained.

In this case, when an attention pattern of (0, 0, 1) corresponds todetermination group patterns GP1, GP2, GP3, GP4, and GP5 in a groupdetermination table, an attention pattern of (0, 0, 2) corresponds tothe determination group patterns GP1 GP2, GP3, GP4, and GP5.

In other words, in the second case, each set of attention patterns whichare made the same by rotation or inversion is represented by arepresentative attention pattern, statistics is obtained forrepresentative attention patterns, a group-pattern determination tableis generated from the statistics for the representative attentionpatterns, and finally, a group determination table is generated for allattention patterns.

With the use of such a method, it is expected that individuality isavoided and highly reliable determination is made with a relativelysmall number of samples.

A detailed procedure for generating a group determination table in thesecond case will be described below.

Prior to the generation of a group determination table based onstatistics, a conversion table for converting actual attention patternsand actual group patterns to representative attention patterns andrepresentative group patterns is generated in advance.

In the same way as in the first case, a condition in which the number ofconference participants is three is taken as an example. Anattention-pattern conversion table shown in FIG. 41, a group conversiontable shown in FIG. 42, and a group inverted-conversion table shown inFIG. 45 need to be prepared in advance.

The attention-pattern conversion table shown in FIG. 41 shows whichrepresentative pattern serves as a representative of an attentionpattern, and which conversion method is used for converting an attentionpattern to a representative attention pattern.

As a representative attention pattern, an attention pattern having thesmallest number is selected when the attention-destination numbers ofthe conference participants HM1, HM2, and HM3 are regarded as a thirddigit, a second digit, and a first digit, respectively, to form adecimal number.

Conversions are expressed by whether inversion is performed and thenumber of rotations under a rule in which inversion is performed firstand then rotation is performed.

FIG. 46A and FIG. 46B show the axis of inversion and the direction ofrotation. Examples of inversion and rotation are shown in FIG. 46A andFIG. 46B.

In the attention-pattern conversion table shown in FIG. 41, whetherinversion is performed is indicated by “0,” which shows no inversion,and “1,” which shows that inversion is performed.

The number of rotations means the number of rotations performed in thedirection shown in FIG. 46B.

The group conversion table shown in FIG. 42 indicates that each grouppattern is converted to which group pattern when the above-describedinversion and rotation are performed.

More specifically, group patterns obtained when neither inversion norrotation is applied to each group pattern, when rotation is appliedonce, when rotation is applied twice, when only inversion is applied,when inversion is applied and then rotation is applied once, and wheninversion is applied and then rotation is applied twice are shown in thetable.

When attention patterns in samples are converted to representativeattention patterns, this table is used for obtaining group patternscorresponding to group patterns in samples which match the conversion.Details will be described later.

The group-pattern inverted-conversion table shown in FIG. 45 indicates agroup pattern obtained when the inverted conversion of a specifiedconversion is applied to each group pattern, and is used for generatingthe group determination table shown in FIG. 40 from arepresentative-group determination table shown in FIG. 44, describedlater.

A device structure and a generation procedure for generating the groupdetermination table shown in FIG. 40 from these tables will be describednext.

FIG. 47 shows functional blocks of a device for generating the groupdetermination table shown in FIG. 40.

Each block can be implemented by either software or hardware. Agroup-determination-table generating device having the functional blocksshown in FIG. 47 may be built, for example, in the seating-orderdetermination device GJD, or may be implemented, for example, by apersonal computer which is a separate device from the seating-orderdetermination device GJD. In either case, when the seating-orderdetermination device GJD is finally made to hold a generated groupdetermination table, if it receives attention-degree information fromthe teleconference devices TCD, it determines the group corresponding toan attention pattern of the conference participants HM1 to HM3 byreferring to the group determination table and generates seating-orderinformation according to the group determination.

The group-determination-table generating device includes, as shown inFIG. 47, information obtaining means 301, information conversion means302, representative-frequency-table generating means 303,representative-group-determination-table generating means 304,representative-group determination means 305, group inverted-conversionmeans 306, and group-determination-table generating means 307.

The information obtaining means 301 obtains attention patterns and grouppatterns as samples obtained in an experiment.

The information conversion means 302 uses the pattern conversion tableshown in FIG. 41 and the group conversion table shown in FIG. 42 toconvert the attention patterns and the group patterns obtained by theinformation obtaining means 301.

The representative-frequency-table generating means 303 generates arepresentative frequency table like that shown in FIG. 43 according torepresentative attention patterns and representative group patterns.

The representative-group-determination-table generating means 304generates a representative group determination table like that shown inFIG. 44 according to the representative frequency table.

The representative-group determination means 305 searches therepresentative-group determination table shown in FIG. 44 to determinethe representative group pattern corresponding to a representativeattention pattern.

The group inverted-conversion means 306 inverted-converts representativegroup patterns to group patterns.

The group-determination-table generating means 307 generates the groupdetermination table shown in FIG. 40 from group patterns and attentionpatterns.

In FIG. 47, solid lines with arrows indicate a flow of generating therepresentative-group determination table shown in FIG. 44, and dottedlines with arrows indicate a flow of generating the group determinationtable shown in FIG. 40 from the representative-group determinationtable.

The group-determination-table generating device having such a structurefirst generates a representative frequency table such as that shown inFIG. 43.

All attention patterns and group patterns in samples are first convertedto representative attention patterns and to group patterns (hereinafter,for convenience, called representative group patterns) to whichconversion matching the conversion from the attention patterns to therepresentative attention patterns is applied, and the frequencies of therepresentative group patterns for each representative attention patternare indicated in a representative frequency table.

FIG. 48 shows processing for generating the representative frequencytable.

In FIG. 48, “registration to representative frequency table” means thatthe frequency of a representative attention pattern and a correspondingrepresentative group pattern is incremented by one.

In step S110, the information obtaining means 301 obtains a first sampleof an attention pattern and a group pattern. Then, in the next stepS111, the information conversion means 302 uses the attention-patternconversion table to search for the representative attention patterncorresponding to the attention pattern obtained as the sample and aconversion method to the representative attention pattern.

In the next step S112, the information conversion means 302 uses thegroup conversion table shown in FIG. 42 to convert the group pattern toa representative group pattern according to the conversion methodsearched for in step S111.

With these steps S111 and S112, the representative attention pattern andthe representative group pattern are obtained for the one obtainedsample.

In the next step S113, the representative-frequency-table generatingmeans 303 registers the obtained representative attention pattern andthe obtained representative group pattern into the representativefrequency table shown in FIG. 43. In other words, the cell correspondingto the current sample is incremented by one.

In step S114, it is determined whether the processing has been finishedfor all samples. When the processing has not yet been finished, theinformation obtaining means 301 obtains the next sample of an attentionpattern and a group pattern in step S115, the processing returns to stepS111, and the same processes as described above are performed.

Therefore, when the processes of steps S111, S112, and S113 have beenfinished for all samples, the representative frequency table shown inFIG. 43 has been completed.

Next, the representative-group determination table shown in FIG. 44 isgenerated from the representative frequency table generated as describedabove.

The representative-group determination table indicates therepresentative group pattern corresponding to each representativeattention pattern. A representative attention pattern in each entry(row) is the same as that shown in the representative frequency table.

The representative-group determination table is generated byregistering, as the representative group pattern corresponding to eachrepresentative attention pattern, the representative group patternhaving the highest frequency for each representative attention patternamong representative group patterns. FIG. 49 shows the specificprocedure of generating the table.

The representative-group-determination-table generating means 304obtains a first entry of the representative frequency table generated bythe representative-frequency-table generating means 303 in step S120. Instep S121, the representative-group-determination-table generating means304 determines the representative-group pattern having the highestfrequency for the representative attention group in the obtained entry,and registers it to the representative-group determination table.

In the representative frequency table shown in FIG. 43, for example, thefrequencies of representative group patterns are indicated for arepresentative attention pattern of (0, 0, 0) in a first entry. Therepresentative group pattern GP1 has the highest frequency. Therefore,as shown in a first row of the representative-group determination tableshown in FIG. 44, the representative group pattern GP1 is registeredcorrespondingly to a representative attention pattern of (0, 0, 0).

In step S122, it is determined whether the processing has been finishedfor all entries in the representative frequency table. When theprocessing has not yet been finished, the next entry is obtained in stepS123 and the processing returns to step S121.

When the process of step S121 is finished for all entries, theprocessing is finished at step S122.

With the above-described processing, the representative-groupdetermination table shown in FIG. 44 is generated from therepresentative frequency table shown in FIG. 43. The processingperformed so far corresponds to the flow indicated by the solid lines inFIG. 47.

The group determination table shown in FIG. 40 is finally generated fromthe representative frequency table.

The group determination table includes all attention patterns. Togenerate the group determination table, the group pattern having the“relationship between the representative attention pattern and therepresentative group pattern registered therefor” corresponding to eachattention pattern in the representative-group determination table, whichrelationship matches the “relationship between the attention pattern andthe group pattern registered therefor,” is registered.

FIG. 50 shows a specific procedure for generating the groupdetermination table. (The procedure corresponds to the flow indicated bythe dotted lines in FIG. 47).

After the representative-group determination table is generated asdescribed above, in step S130 shown in FIG. 50, the informationobtaining means 301 obtains a first attention pattern. In step S131, theinformation conversion means 302 uses the attention-pattern conversiontable to search for a representative attention group and a conversionmethod to the representative attention pattern for the obtainedattention pattern. The result of searching is passed to therepresentative-pattern determination means 305.

In the next step S132, the representative-group determination means 305uses the representative-group determination table to search for therepresentative group pattern corresponding to the representativeattention pattern passed from the information conversion means 302. Theresult of searching is passed to the group inverted-conversion means 306as the representative group pattern.

In step S133, the group inverted-conversion means 306 uses the groupinverted-conversion table shown in FIG. 43 to inverted-converts therepresentative group pattern passed from the representative-groupdetermination means 305 to a group pattern.

In step S134, the group-determination-table generating means 307registers an entry formed of the attention pattern obtained by theinformation obtaining means 301 as described above, and the grouppattern sent from the group inverted-conversion means 306 into the groupdetermination table.

In step S135, it is determined whether the processing has been finishedfor all attention patterns. When the processing has not yet beenfinished, the next attention pattern is obtained in step S136, and theprocessing returns to step S131.

When registration has been made to the group determination table for allattention patterns, the processing is finished at step S135.

With the above-described processing, the group determination table shownin FIG. 40 is generated.

A group determination table used for group determination is generated asdescribed above. The seating-order determination device GJD performsgrouping with the use of a group determination table to generateseating-order information.

9. Seating-Order Determination Operation not Through Grouping in theSeating-Order Determination Device

So far, a case has been described in which the seating-orderdetermination device GJD first performs group determination and thendetermines a seating order according to the result of groupdetermination. A seating order can be determined without performinggroup determination. An example case will be described below.

The seating-order determination device GJD holds an information requestdegree Rij indicating a degree at which each conference participant HMiwants the information of another conference participant HMj. It isconsidered, for example, that the information request degree is high foran attention destination at a point of time close to the current time.When a conference participant HMi paid attention to conferenceparticipants HM2 and HM5 in the past and currently pays attention aconference participant HM3, for example, the information request degreesof the conference participant HMi for these conference participants areset to areas Ri2, Ri5, and Ri3 located under a curve shown in FIG. 52.The curve shown in the figure is based on an exponential function for aconstant K larger than zero and smaller than one.

More specifically, the seating-order determination device GJD checkswhether each conference participant HMi pays attention to anotherconference participant HMj at a constant interval, and sets a variableAij indicating whether attention is paid to “1” when the conferenceparticipant HMi pays attention to the conference participant HMj, andsets it to “0” when the conference participant HMi does not payattention to the conference participant HMj.

When attention checking is finished, the seating-order determinationdevice GJD calculates the information request degree of the conferenceparticipant HMi for the conference participant HMj at a time “It” by thefollowing expression (1).

Rij(t)=KRij(t−1)+Aij(t)  (1)

where, K is an attenuating coefficient.

Then, the seating-order determination device GJD calculates an overallsatisfaction degree Sm of the entire conference participants for each ofall possible seating-order candidates (seating-order number m). Thefollowing expression (2) is used.

$\begin{matrix}{S_{m} = {\sum\limits_{i}{\sum\limits_{j}{{Wmij} \cdot {Rij}}}}} & (2)\end{matrix}$

where, Wmij is a satisfaction-degree weighting coefficient determined inadvance for each information request degree in each seating order, andheld, for example, in a satisfaction-degree weighting table shown inFIG. 51.

The table shown in FIG. 51 shows a case in which the number ofconference participants is six. Characters A to F correspond to seatnumbers, and numerals 1 to 6 shown therebelow correspond to conferenceparticipants HM1 to HM6.

Since the relative positional relationships among conferenceparticipants are meaningful in a seating order, when it is specified,for example, that a conference participant HM1 is always assigned to aseat A, the number of seating orders is equal to the number of thepermutations of five things, which is 120, and a seating-order numberranges from 1 to 120.

FIG. 53 shows an example arrangement of seats A to F.

The satisfaction-degree weight Wmij shown in FIG. 51 is set, forexample, larger when the distance between HMi and HMj is closer in aseating order “m.” More specifically, for example, the reciprocal1/Dij_(—)1 of the number Dij_(—)1 indicating that HMj is located at theDij_(—)1-th position from HMi, or the reciprocal l/Dij_(—)2 of a numberDij_(—)2 indicating the distance between HMi and HMj when the distancebetween adjacent seats is set to “1” can be used as thesatisfaction-degree weight.

In a seating order shown in FIG. 54, D16 _(—)1 is 2, and D16_2 is√{square root over (3)}.

The seating-order determination device GJD determines the seating ordercorresponding to the maximum satisfaction degree Sm as a result ofcalculation. When there is a plurality of the maximum satisfactiondegrees Sm, the seating order which makes the sum of the distances ofmovements required for the other conference participants, viewed fromeach conference participant smallest, or the seating order which has thesmallest seating-order number can be selected.

10. Attention-Degree-Information Generating Operation in aTeleconference Device

Various operations for dynamically changing a seating order according tothe attention degrees of conference participants have been described. Aspecific processing for detecting a direction (direction toward any ofthe monitor devices MD2 to MDn or another direction) in which aconference participant HM1 pays attention, according to image data sent,for example, from the camera of the monitor device MDm disposed at thefront of the conference participant HM1 in theattention-degree-information generating section JB1 shown in FIG. 2 willbe described.

As a first example of the processing for detecting a direction in whichthe conference participant HM1 pays attention, to be performed in theattention-degree-information generating section JB1 of theteleconference device TCD1 according to the present embodiment, thedetection (sight-line detection) of the lines of sight of the conferenceparticipant HM1 can be taken.

FIG. 55 is a flowchart of sight-line detection processing in theattention-degree-information generating section JB1.

In FIG. 55, the attention-degree-information generating section JB1receives image data captured by the camera provided, for example, forthe monitor device MDm disposed at the front of the conferenceparticipant HM1 in step S11. In the next step S12, theattention-degree-information generating section JB1 uses the colorinformation of the sent image to detect the outlines of both eyes of theconference participant HM1 in a facial image. More specifically, theattention-degree-information generating section JB1 extracts colorareas, such as skin, whites, and irises, by using the color informationof the sent image, and obtains, for example, the boundaries of theextracted color areas to detect the outline E of the right eye and thatof the left eye as shown in FIG. 56. FIG. 56 indicates only one eye.

Then, in step S13, the attention-degree-information generating sectionJB1 obtains the positions of the leftmost point F1 and the rightmostpoint F2 of the right eye and those of the left eye according to theoutlines E of both eyes obtained in step S12, determines a search areaNE for searching for the nostrils, as shown in FIG. 57, with thepositions of the rightmost and leftmost points F2 and F1 of the rightand left eyes being used as references, and detects the positions of thenostrils NH in the search area NE. More specifically, theattention-degree-information generating section JB1 obtains a line Mwhich makes the center Q of gravity of the sets of pixels constitutingthe outlines E of the right and left eyes and the secondary moment(inertia for the line) of the sets of pixels constituting the outlines Esmallest; obtains pixels one each in the right and left directions,located at the largest distances L1 and L2 from the center Q of gravityin the directions of line M; and obtains the pixels as the rightmost andleftmost points F2 and F1, as shown in FIG. 56.

Next, the attention-degree-information generating section JB1 uses thepositions of the rightmost and leftmost points F2 and F1 of the rightand left eyes, obtained as described above, as references and determinesthe search area NE for searching for the nostrils in the lower directionfrom the rightmost and leftmost points F2 and F1, as shown in FIG. 57.Since the images of the nostrils NH are darker than that of the otherparts, the attention-degree-information generating section JB1 detectslow-luminance image areas as the positions of the nostrils NH in thesearch area NE.

Then, in step S14, the attention-degree-information generating sectionJB1 assumes the central positions ECs of the eyeballs EBs and the radius“r” of the eyeballs EBs according to the geometrical positionalrelationships among the positions of the rightmost and leftmost pointsF2 and F1 of the right eye, those of the rightmost and leftmost pointsF2 and F1 of the left eye, and those of the nostrils NH, as shown inFIG. 58.

In step S15, the attention-degree-information generating section JB1uses the luminance information of the image in the outline E of theright eye and that in the outline E of the left eye to detect thecentral positions EAC of the pupils EA.

In step S16, the attention-degree-information generating section JB1calculate vectors EV connecting between the central positions EC of theeyeballs EB detected in step S14 and the central positions EAC of thepupils EA detected in step S15, regards the obtained vectors EVs as thelines of sight, and determines the directions in which the vectors EVsare directed, namely, determines the monitor to which the vectors EVsare directed among the monitor devices MD2 to MDn.

With the foregoing flow, the attention-degree-information generatingsection JB1 detects the lines of sight of the conference participantHM1.

A line M which makes the secondary moment of the set of pixels, such asthat of pixels constituting the outline E can be obtained, for example,by the following calculation.

A straight line M indicated by an expression (3), as shown in FIG. 60,will be taken as an example.

x sin θ−y cos θ+ρ=0  (3)

The secondary moment for the straight line M can be indicated by anexpression (4) where Ri indicates the distance between the straight lineM and each point (xi, yi) of the set of pixels constituting the outlineE.

$\begin{matrix}{m = {{\sum\limits_{i}{Ri}^{2}} = {\sum\limits_{i}( {{x_{i}\sin \; \theta} - {y_{i}\cos \; \theta} + \rho} )^{2}}}} & (4)\end{matrix}$

The straight line M which makes the secondary moment smallest is thestraight line M which makes “m” in the expression (4) minimum. To make“m” in the expression (4) minimum, θ and ρ satisfying the followingconditions (5) and (6) are used as those in the expression (4).

θ: sin 2θ=b/(b ²+(a−c)²)^(1/2), cos 2 θ=(a−c)/(b ²+(a−c)²)^(1/2)  (5)

ρ: ρ=−x ₀ sin θ+y ₀ cos θ  (6)

The expression (6) (x₀ sin θ+y₀ cos θ+ρ=0) indicates that the linepasses through the center of gravity of the set of pixels.

In the expressions (5) and (6), “a,” “b,” and “c” are indicated byexpressions (7), (8), and (9), respectively. (x₀, y₀) indicates thecoordinates of the center of gravity of the set of pixels.

$\begin{matrix}{a = {\sum\limits_{i}( {x_{i} - x_{0}} )^{2}}} & (7) \\{b = {2{\sum\limits_{i}{( {x_{i} - x_{0}} )( {y_{i} - y_{0}} )}}}} & (8) \\{c = {\sum\limits_{i}( {y_{i} - y_{0}} )^{2}}} & (9)\end{matrix}$

As a second example of the processing for detecting a direction in whichthe conference participant HM1 pays attention, to be performed in theattention-degree-information generating section JB1 of theteleconference device TCD1 according to the present embodiment, thedetection of the face direction of the conference participant HM1 can betaken, which will be described below.

FIG. 61 shows a flowchart of processing for detecting a face directionin the attention-degree-information generating section JB1.

In FIG. 61, the attention-degree-information generating section JB1receives original image data, such as that shown in FIG. 62A and FIG.62B, of the face of the conference participant HM1 from the monitordevice MDm disposed at the front of the conference participant HM1 instep S21. In the next step S22, the attention-degree-informationgenerating section JB1 uses the color information of the received faceimages to detect a skin area and a hair area. More specifically, theattention-degree-information generating section JB1 extracts skin-colorand hair-color areas by using the color information of the received faceimages, and detects a skin area “se” and a hair area “he” by theextracted color areas, as shown in FIG. 63A and FIG. 63B.

In the next step S23, the attention-degree-information generatingsection JB1 specifies frames for detecting the center “fg” of gravity ofthe total area “fe (=“se”+“he”)” of the skin area “se” and the hair area“he” and the center “sq” of gravity of the skin area “se,” as shown inFIG. 64A and FIG. 64B. The frames are specified, for example, by settingzones in the vertical direction in the images. More specifically, forexample, the upper end “re” of the total area “fe” of the hair area “he”and the skin area “se” is used as a reference, and a zone is specifiedbetween a point a length “const_a” below the upper end “re” and a pointa length “const_a”+“const_b” below the upper end “re.”

Then, in step S24, the attention-degree-information generating sectionJB1 obtains the center “fg” of gravity of the total area “fe” of theskin area “se” and the hair area “he” and the center “sq” of gravity ofthe skin area “se” within the frames specified in step s23. In asubsequent process, both the horizontal components and verticalcomponents of these centers of gravity can be used, or either thehorizontal components or the vertical components of the centers ofgravity can be used. As an example, a case in which only the horizontalcomponents of the centers of gravity are used is taken, and will bedescribed below.

In step S24, the attention-degree-information generating section JB1obtains the center “fg” of gravity of the total area “fe” of the skinarea “se” and the hair area “he” and the center “sq” of gravity of theskin area “se.” In step S25, the attention-degree-information generatingsection JB1 calculates the difference obtained by subtracting the center“fg” of gravity of the total area “fe” of the skin area “se” and thehair area “he” from the center “sq” of gravity of the skin area “se.”

Then, in step S26, the attention-degree-information generating sectionJB1 detects a face direction by using the difference obtained in stepS25. More specifically, either of the following two methods are, forexample, used to detect a face direction by using the difference. It isassumed that X indicates a difference, Y indicates a face-directionangle, and the angle of the face of the conference participant HM1 isset to 0 degrees when the conference participant HM1 is directed to thecamera of the monitor device MDm. In one method used in step S26, priorto face-direction detection processing, data for the difference X andthe face-direction angle Y is obtained in advance; the face-directionangle Y corresponding to the difference X is obtained, for example, asthe average; their relationship is obtained as shown in FIG. 65; and theface-direction angle Y is obtained from the difference X obtained instep S25, according to the relationship shown in FIG. 65. In anothermethod used in step S26, the face-direction angle Y is obtained from thefollowing expression (10) by using the difference X obtained in stepS25.

Y=α sin(X)  (10)

With the above flow, the attention-degree-information generating sectionJB1 detects the face direction of the conference participant HM1.

In still another method for detecting the direction in which theconference participant HM1 is directed, for example, an infrared ray isemitted to the face of the conference participant HM1; an infrared rayreflected from the face of the conference participant HM1 is received toform an image; and the face direction is detected from the image.

11. Structure of Monitor Device

An example specific structure of each of the monitor devices MD2 to MDnin the structure shown in FIG. 2 will be described next by referring toFIG. 66 and FIG. 67. FIG. 66 is an outlined internal view of a monitordevice MD, viewed from a side. FIG. 67 is an outlined elevation of themonitor device MD.

In the following description, for simplicity, a case is taken as anexample, in which information related to conference participants HM1 toHMn is displayed on monitor devices MD1 to MDn in teleconference devicesTCD1 to TCDn.

In the present embodiment, each of the monitor devices MD2 to MDn isprovided, as shown in FIG. 66 and FIG. 67, with a cabinet 10; a speaker13 disposed at the front (front of the monitor device MD) of the cabinet10; a display section 15 disposed such that a screen 14 is directed in apredetermined direction (upper direction in the case shown in FIG. 66);a half mirror 12 for reflecting light emitted from the screen 14 of thedisplay section 15 towards the front of the monitor device MD along aone-dot chain line BO in the figure and for passing light incident fromthe front of the monitor device MD along a two-dot chain line BI in thefigure; and a camera 16 (such as a video camera) supported by asupporting section 18 behind the half mirror 12. On the upper surface ofthe cabinet 10 in the monitor device MD, for example, a microphone 11supported by a supporting section 17 is also provided.

The microphone 11 may be provided, for example, only for the monitordevice (monitor device MDm in the case shown in FIG. 2) disposed at thefront of the conference participant HM1 among the monitor devices MD2 toMDn.

The camera 16 of each of the monitor devices MD2 to MDn receivesincident light (such as an optical image of the conference participantHM1) passing through the half mirror 12 along the two-dot chain line BIin FIG. 66, and converts it to image data. The image data output fromthe camera 16 is sent to the information transmitting and receivingsection TRB1 of the signal processing device SPD1, and then sent to theteleconference devices TCD2 to TCDn through the communication networkNT. The image data output from the camera 16, for example, of themonitor device MDm disposed at the front of the conference participantHM1 among the monitor devices MD2 to MDn is also sent to theattention-degree-information generating section JB1 of the signalprocessing device SPD1 and is used for detecting lines of sight or aface direction when attention-degree information is generated, asdescribed above.

The microphone 11 of each of the monitor devices MD2 to MDn convertssound, such as surrounding sound of the teleconference device TCD1 andwhat the conference participant HM1 says, to audio data. The audio dataoutput from the microphone 11 is sent to the information transmittingand receiving section TRB1 of the signal processing device SPD1, andthen, sent to the teleconference device TCD2 to TCDn through thecommunication network NT.

On the screen 14 of the display section 15 in the monitor device MD2among the monitor devices MD2 to MDn, an image based on image data (thatof the conference participant HM2 and the surroundings) captured by thecamera 16 of the monitor device. MD1 provided correspondingly to theconference participant HM1 in the teleconference device TCD2 and sentthrough the communication network NT is displayed. From the speaker 13of the monitor device MD2, sound based on audio data (that of what theconference participant HM2 says) captured by the microphone 11 of themonitor device MD1 provided correspondingly to the conferenceparticipant HM1 in the teleconference device TCD2 and sent through thecommunication network NT is reproduced. In the same way, on the screen14 of the display section 15 in the monitor device MD3, an image basedon image data captured by the camera 16 of the monitor device MD1provided correspondingly to the conference participant HM1 in theteleconference device TCD3 and sent through the communication network NTis displayed. From the speaker 13 of the monitor device MD3, sound basedon audio data captured by the microphone 11 of the monitor device MD1provided correspondingly to the conference participant HM1 in theteleconference device TCD3 and sent through the communication network NTis emitted. The situation is the same for the other monitor devices MD.An image sent from the corresponding teleconference device is displayedand sound is emitted.

In each of the monitor devices MD2 to MDn in the present embodiment, asshown in FIG. 66, since light emitted from the screen 14 of the displaysection 15 is reflected by the half mirror 12 in the direction indicatedby the one-dot chain line BO towards the conference participant HM1, aface image and the like of the conference participant HM located at theother side is displayed on the screen 14 of the display section 15 as amirror image, which is reflected by the half mirror 12 to be in acorrect state. In FIG. 67, RV indicates an image (a virtual image of theconference participant HM at the other side) obtained when a mirrorimage of the conference participant HM located at the other side,displayed on the screen 14 of the display section 15 is reflected by thehalf mirror 12.

When a mirror image of the conference participant at the other side isdisplayed on the screen 14 of the display section 15 in a monitor deviceMD in the present embodiment, the positions of the eyes in the virtualimage, which are optically conjugate with those of the eyes in themirror image are displayed so as to almost match the principal point ofthe lens of the camera 16 through the half mirror 12. Therefore, thelines of sight of the conference participant HM1 and those of theconference participant at the other side match.

More specifically, a case in which the conference participant HM1 seesthe monitor MDm (an image of the conference participant HMm) in theteleconference device TCD1 and the conference participant HMm sees themonitor device MD1 (an image of the conference participant HM1) in theteleconference device TCDm is taken as an example and will be described.In this case, a mirror image of the face or the like of the conferenceparticipant HMM is displayed on the screen 14 of the display section 15in the monitor device MDm in the teleconference device TCD1; and thecamera 16 of the monitor device MDm captures an image of the conferenceparticipant HM1 who is directed to the monitor device MDm and sendsimage data to the teleconference device TCDm and others. A mirror imageof the face or the like of the conference participant HM1 is displayedon the screen 14 of the display section 15 in the monitor device MD1 inthe teleconference device TCDm; and the camera 16 of the monitor deviceMD1 captures an image of the conference participant HMm who is directedto the monitor device MD1 and sends image data to the teleconferencedevice TCD1 and others.

In this condition, at the teleconference device TCD1, when a mirrorimage of the conference participant HMm at the other side is displayedon the screen 14 of the display section 15 in the monitor device MDm,the positions of the eyes in the virtual image, which are opticallyconjugate with those of the eyes in the mirror image are displayed so asto almost match the principal point of the lens of the camera 16. At thesame time, at the teleconference device TCDm, when a mirror image of theconference participant HM1 at the other side is displayed on the screen14 of the display section 15 in the monitor device MD1, the positions ofthe eyes in the virtual image, which are optically conjugate with thoseof the eyes in the mirror image are displayed so as to almost match theprincipal point of the lens of the camera 16. Therefore, at theteleconference device TCD1, the lines of sight of the conferenceparticipant HM1 match those of the virtual image of the conferenceparticipant HMm at the other side. At the teleconference device TCDm,the lines of sight of the conference participant HMm match those of thevirtual image of the conference participant HM1 at the other side.

In general conventional teleconference systems, conference participantsdo not see virtual images made by half mirrors but directly see images(real images) displayed on the screens of display sections. In addition,cameras are disposed above or below, or at the right or left of thescreens of the display sections in their vicinities. Therefore, ingeneral conventional teleconference systems, the lines of sight ofconference participants are directed to images (real images) displayedon the screens of the display sections, and are not directed to thelenses of the cameras. Consequently, the lines of sight of theconference participant at the other side, displayed on the screen of adisplay section does not seem to be directed to you. Unlike the presentembodiment, it is impossible to perform conversation while the lines ofyour sight match those of the conference participant at the other side.

In contrast, in the teleconference system according to the presentembodiment, when the monitor device of each teleconference device TCDhas the structure shown in FIG. 66 and FIG. 67, a conference participantcan perform conversation with the conference participant at the otherside while they see their eyes each other, namely, the lines of theirsight match.

In the present embodiment, when a plurality of monitor devices MD in ateleconference device TCD are disposed, as shown in FIG. 2, as if theconference participants HM2 to HMn located at the teleconference devicesTCD2 to TCDn and the conference participant HM1 sat around a table,namely, when the plurality of monitor devices are disposed such that therelative positional relationships among the conference participants HM2to HMn at the places where the teleconference devices TCD2 to TCDn aredisposed are maintained, not only the lines of sight match between theconference participant HM1 and the conference participant at the otherside, but also the conference participant HM1 understands whom the otherconference participants HM are directed to.

12. Example Structure of Each Device

FIG. 68 shows an actual example device structure which can be used forthe signal processing device SPD of each teleconference device TCD orthe seating-order determination device GJD, in a teleconference systemaccording to an embodiment of the present invention. These devices canbe implemented, for example, by personal computers. Thegroup-determination-table generating device like that shown in FIG. 47can also be implemented by the following device structure.

The structure shown in FIG. 68 includes a CPU 100 for controlling eachsection; a ROM 101 for storing basic input and output systems (BIOS) andvarious initial values; a RAM 102 for tentatively storing variousprograms, data, and data obtained during calculation; a hard-disk drive(HDD) 104 for driving a hard disk for storing an operating system (OS),various application programs (computer programs), and other data; aremovable-medium drive 105 for driving a removable medium 106, such as aCD-ROM, a CD-R, a CD-RW, a DVD-ROM, a DVD-RAM, a DVD-R, a DVD-RW, aremovable hard disk, and a semiconductor memory; and a communicationinterface section 103 for connecting to an external communicationnetwork (the communication network NT), such as an ISDN, a commercialtelephone line, a cable-TV line, and a digital communication-satelliteline, and for connecting to an external bus, such as that conforming tothe IEEE-1394 standard or a USB, and to various external connectionterminals.

The structure shown in FIG. 68 can further include, for example, aninput operation device, such as a mouse or a keyboard, operated by theuser and a monitor for displaying information, although they are notshown.

An application program for implementing the functions of the signalprocessing device SPD in the teleconference system according to thepresent embodiment described above, especially theattention-degree-information generating function in theattention-degree-information generating section JB1 and the informationmanipulation and distribution function in the information manipulationand distribution section PB, or the group determination processing, theseating-order determination processing, and theseating-order-information generating function is provided by theremovable medium 106 or by communication through the communicationinterface section 103.

The application program provided by the removable medium 106 or by thecommunication interface section 103 is stored in the hard disk of theHDD 104, is read from the hard disk of the HDD 104, and tentativelystored in the RAM 102. The CPU 100 executes various operations in theteleconference system according to the present embodiment according tothe application program tentatively stored in the RAM 102.

FIG. 69 shows another example structure of the teleconference deviceTCD1.

In the example structure shown in FIG. 69, as display means fordisplaying images of conference participants HM2 to HMn inteleconference devices TCD2 to TCDn, the monitor devices MD2 to MDn,such as those shown in FIG. 2, corresponding to the teleconferencedevices TCD2 to TCDn (the conference participants HM2 to HMn) are notprovided, but, for example, one bent screen 31 is provided and imagesare displayed on the screen 31, for example, by a projector.

In the example structure shown in FIG. 69, images of the conferenceparticipants HM2 to HMn are displayed on the screen 31 as if theconference participant HM1 and the other conference participants HM2 toHMn sat around a table for a conference.

A camera 34 and a microphone 35 are disposed, for example, at the frontof the conference participant HM1. Image data of the conferenceparticipant HM1, captured by the camera 34 and audio data of theconference participant HM1, captured by the microphone 35 are sent tothe other teleconference devices TCD2 to TCDn through the communicationnetwork NT. In the example structure shown in FIG. 69, the image data ofthe conference participant HM1, captured by the camera 34 is also sentto the attention-degree-information generating section JB1 of the signalprocessing device SPD1.

Audio data of conference participants HM2 to HMn, sent from the otherteleconference devices TCD2 to TCDn are controlled such that individualsound images are formed in the vicinities of images of the conferenceparticipants HM2 to HMn, displayed on the screen 31, and are sent tospeakers 32 and 33 disposed at the right and left of the screen 31 andemitted. With this control, the positions of images of the conferenceparticipants HM2 to HMn, displayed on the screen 31 almost match thoseof locations where the voices (sound) of the conference participants HM2to HMn are heard.

In the present embodiment, the attention-degree-information generatingsection JB is disposed in each of the signal processing devices SPD1 toSPDn of the teleconference devices TCD1 to TCDn. Onattention-degree-information generating section JB may be independentlyprovided on the communication network NT.

In the present embodiment, as shown in FIG. 2, the monitor devices MD2to MDn are separated from the signal processing device SPD1. Each or oneof the monitor devices MD2 to MDn can have the function of the signalprocessing device.

In the present embodiment, as shown in FIG. 1, the seating-orderdetermination device GJD is independently connected to the communicationnetwork NT. Each or one of the teleconference devices TCD1 to TCDn canhave the function of the seating-order determination device.

In the present embodiment, as examples for detecting a direction, thelines of sight and a face direction are separately detected. They can bedetected at the same time.

In the present embodiment, one conference participant belongs to onlyone group at each point of time. It is also possible that a plurality ofgroups is defined, such as a group to which a conference participantmainly belongs and a group in which a conference participant does notgive opinions but from which the conference participant wants to obtaininformation; each conference participant is allowed to belong to aplurality of groups; and a seating order is determined according towhich group each of the conference participants at the other sidesbelongs to.

As described above, according to the teleconference system of thepresent embodiment, even when a plurality of conference participants sayat the same time, it is easier for a conference participant to listen toa speech in a group which the conference participant belongs to, and itis also easier to see images. Therefore, the teleconference systemprovides each conference participant with comfort and satisfaction withinformation.

1-36. (canceled)
 37. A group-determination-table generating device forgenerating a group determination table used for grouping informationsent from each communication device according to the degree of attentionof the user of each communication device in a communication systemhaving at least three communication devices, comprising: statisticsmeans for reading attention-degree patterns indicating the degree ofattention which the user of each communication device pays toinformation sent from other communication devices and group structurepatterns indicating the group state of each user, and for collectingstatistics on the attention-degree patterns and the group structurepatterns; determination means for determining the correspondence betweeneach attention-degree pattern and one of the group structure patternsfrom the statistics obtained by the statistics means; anddetermination-table generating means for generating a groupdetermination table indicating attention-degree patterns and groupstructure patterns for which the correspondence is determined by thedetermination means.
 38. A group-determination-table generating deviceaccording to claim 37, wherein the statistics means collects statisticsby regarding a plurality of attention-degree patterns substantially thesame as one representative attention-degree pattern; and thedetermination means determines the representative group patterncorresponding to the representative attention-degree pattern, and then,according to the determination, determines the correspondence betweeneach attention-degree pattern and one of the group structure patterns.39-59. (canceled)
 60. A group-determination-table generating method fora group-determination-table generating device for generating a groupdetermination table used for grouping information sent from eachcommunication device according to the degree of attention of the user ofeach communication device in a communication system having at leastthree communication devices, comprising: a statistics step of readingattention-degree patterns indicating the degree of attention which theuser of each communication device pays to information sent from othercommunication devices and group structure patterns indicating the groupstate of each user, and of collecting statistics on the attention-degreepatterns and the group structure patterns; a determination step ofdetermining the correspondence between each attention-degree pattern andone of the group structure patterns from the statistics obtained in thestatistics step; and a determination-table generating step of generatinga group determination table indicating attention-degree patterns andgroup structure patterns for which the correspondence is determined inthe determination step.
 61. A group-determination-table generatingmethod according to claim 60, wherein, in the statistics step,statistics is collected by regarding a plurality of attention-degreepatterns substantially the same as one representative attention-degreepattern; and in the determination step, the representative group patterncorresponding to the representative attention-degree pattern isdetermined, and then, according to the determination, the correspondencebetween each attention-degree pattern and one of the group structurepatterns is determined. 62-82. (canceled)
 83. A computer-readablerecording medium for storing a processing program related togroup-determination-table generation in a group-determination-tablegenerating device for generating a group determination table used forgrouping information sent from each communication device according tothe degree of attention of the user of each communication device in acommunication system having at least three communication devices, theprocessing program controlling a processor to perform the methodcomprising: a statistics step of reading attention-degree patternsindicating the degree of attention which the user of each communicationdevice pays to information sent from other communication devices andgroup structure patterns indicating the group state of each user, and ofcollecting statistics on the attention-degree patterns and the groupstructure patterns; a determination step of determining thecorrespondence between each attention-degree pattern and one of thegroup structure patterns from the statistics obtained in the statisticsstep; and a determination-table generating step of generating a groupdetermination table indicating attention-degree patterns and groupstructure patterns for which the correspondence is determined in thedetermination step.
 84. A computer-readable recording medium accordingto claim 83, wherein, in the statistics step, statistics is collected byregarding a plurality of attention-degree patterns substantially thesame as one representative attention-degree pattern; and in thedetermination step, the representative group pattern corresponding tothe representative attention-degree pattern is determined, and then,according to the determination, the correspondence between eachattention-degree pattern and one of the group structure patterns isdetermined.